2006-04-25 05:22 pm (UTC)
You so slick, girl. Nice work.
2006-04-29 05:36 pm (UTC)
how about HTML retreival
Nice article! I am however interested in a different scenario where the the script places a request to a URL and retrieves the page as a whole so it can later loop through and extract some values. Due to the semi-dynamicness (can I even say that?:) of the remote page, I need it as a whole. The fact that I dont really trust the webmaster drives me to want to have the browser do as less automatic parsing as possible.
Thanks for reading.
If you don't trust the remote server, I don't think that this technique is for you. I did this for a situation in which I was the remote server, and I wanted to give some code to some other people that would allow them to use a service I was providing.
Without knowing more about the situation, I'm not sure if I can help. Good luck.
2006-05-02 12:45 pm (UTC)
Hey this is some cool stuff. Might work well with something I've been playing with, though I think I may have settled on a less complex (!?) method.
Since you are obviously an uber-geek (shhh, is a grand compliment) maybe you could tell me what you think:
Topic: Re-distributing the internet.
Purpose: Certain sites get a massive amount of traffic, but for brief intervals: e.g. a page gets /.ed, offers monthly updates etc. Bandwidth, being a finite resource, means that the site either goes down, or is incredibly slow.
Issue: Current distributed methods require 'significant' effort by the user (and site admin). Bit torrent, for example, requires a seed to be hosted, and the client (user) must download a torrent prog.
Solution?: Assume some site (TinySite.com) hosts a large file (uberfile.7z) which changes every month, which thousands of people try to download at 8pm on the 1st.
Instead of setting the link on the download page statically:
Each user visiting the site might be running this new distributed client, call it ExaNet. But the users don't have to be running the client.
In the java script, there is a simple function which checks to see if a connection to localhost can be established (on the ExaNet port, this would best be done with a hosts file listing). The function tries to load a (small) image from localhost:ExaPort, for instance:
a = "http://localhost:10000/status.jpg";
or if the client is using the hosts file (which it should, else every client on the planet would have to have the same port available!)
a = "http://exanet/status.jpg";
Then, using image.onLoad() and image.onError(), you decide which link to give the user: The standard link for users without ExaNet
or the distrbuted client file for users with:
The ExaNet client then handles the getting of the file over the distributed network; if the file does not exist, it can download it from the actual site.
I believe (am virtually certain) that the client would be able to push the file out to the user without the user having a CLUE that it was being served up from a 'local' source. One issue I do see is that a distributed download is not 'linear', and browsers expect a linear download. So even though you are downloading this 100MB file at a MB/s, the user would only see the linear progression, which could easily be zero if the last chunk to be downloaded by the client was the first chunk in the file.
There are a fat TONNE of security issues involved, but ironically with even a half-assed design something like this could make the internet SAFER, not less so. To whit:
Assume that you require TinyServer.com to have a file in the root directoy called 'ExaInfo.xml'. ExaInfo.xml has a listing of each and every file which TinyServer.com wants 'distributed'. In addition, require that the file listings contains the MD5 of each file. Now, you have restricted TinyServer.com from listing a file from another site (because upon receiving a status check, the ExaNet client will download and parse the ExaInfo file from the root dir, and ignore links to files that aren't defined), as well as ensured that if somehow the file DOES get replaced (e.g. hacked version of ExaClient) by malevolent forces, the MD5 won't match for future users.
I have little idea why I am posting this here :~) aside from the fact that there are some damned smart people that read this journal.
Honestly I'm wondering if it is a good idea or not. I see absolutely zero way of marketing it is a product, so if I do anything with it at all it would be to open-source a demo and hope some more capable than I decide to adopt it. Watcha think? Anyone? :~)
Originally I was thinking 'hey, this would be cool to do using AJAX-like calls!'. Then I woke up. So that's how your post reminded me of the above :~)
For reference, yes, yes I did type all of this into the LiveJournal window ;~)
StarsAreAlsoFire (AOL IM)
Thanks for reading! I hope you don't mind that I reformatted and re-posted your comment.
Let me make sure I understand what you're trying to do here. Your case against BitTorrent seems to be that it requires special, standalone client software. Wouldn't your hypothetical ExaNet application require the same thing? If the idea is that ExaNet does its thing within the browser without any standalone software, then I would think that your first step would be figuring out how to implement a peer-to-peer file sharing app as a browser plugin or extension. (If I misunderstood and that isn't the idea, then what's the benefit of using ExaNet instead of BitTorrent?)
Assuming for a moment that ExaClient would be implemented as a browser extension, I don't think you'd need to do any client-side scripting or remote image loading to tell whether the user has ExaClient or not. Instead, just have ExaClient modify the user-agent string when it's running.
Finally, I'm not sure what you mean by "served up from a 'local' source." I'm guessing you mean "served from a remote source, but arriving by way of the local ExaClient, and in such a way that it looks exactly the same as any other kind of file download." That's something you'd need to deal with while writing your extension. (Personally, I'm not sure I'd want it to look like any other download -- I might want to at least have an idea of what was really happening behind the scenes. But that's just me.)
That's my take on it. I hope I haven't completely mangled the idea you're trying to propose. Best of luck!
2006-05-04 08:18 am (UTC)
Thoughts started to coalesce as a typed, so I started pulling it into a descriptive page:
I still don't know if I really intend to pursue this at all; I greatly fear becoming one of those jackasses that thinks he has a fix for xyz protocol (the most popular being e-mail, though any target of zealotry will do).
Right now I'm still trying to find the time to finish the project that my article grew out of in the first place, so unfortunately I don't have any free time to investigate this further right now. In the meantime, maybe one of those "damned smart people" you mentioned will want to get involved. Good luck.
2006-11-01 10:03 am (UTC)
Google License question
I've scoured the net for someone talking about exactly this. Your article which states 'what if I want to hand off some code to someone else, with the understanding that it will live on their server and connect to mine?' is exactly what I want to be able to do for my clients.
This then begs the question...Is this allowed under googles terms of service?
I know people are charging themselves out to build google maps with their expertise, but what if I want to do the same (allowing my clients to host their own map that's on public display) and have their maps source their data from my server. Can I charge my clients for this without breaking googles terms of service?
This, I've also scoured the net for and can't find the definitive answer.
Your thoughts and or examples of people doing this already would be great.
Thanks for reading!
I'm not sure I understand what you're asking. There are a number of articles out there about connecting to services like Google Maps. Bidynodes, though, is about running a service of your own and making it easy for other people (like your clients) to connect to you. That was my intent, anyway. How would a third-party service like Google Maps come into play?