Using P2P for Legitimate Applications? 50
scum-o asks: "Where I work, we move a lot of large weather data around and there's always a question of whether someone's already found the data that I need to use (many projects use the same data, but it needs to get refreshed several times a day). My brilliant idea was to use a P2P-like network to search for already-existing data and use that in my app (and if none found, go to the original source). My company has a fast network and I'd much rather have my app suck the data from someone else in my company who's already grabbed the data as opposed to pounding on the public ftp server (which is slow and horribly abused each day). Has anyone found any way to use the P2P-network for legitimate reasons other than just file swapping/sharing and stuff? Also, how would I go about this, can I just grab a gnutella API and start searching?"
Gnucleus (Score:3, Informative)
Extend LimeWire (Score:2)
Check out:
limewire.org [limewire.org] and the javadocs [limewire.org].
If you are running Windows (Score:4, Interesting)
But why isn't everyone uploading their data to a central server anyway?
Bitorrent (Score:4, Informative)
Re:Bitorrent (Score:1, Insightful)
Re:Bitorrent (Score:2)
Frankly, this sort of thing is what BitTorrent was designed for.
I hate to sound like a karma whore... (Score:2)
I also became a Robotech fan, resulting in the purchase of a number of DVDs. Technically that's not legitimate, but I would argue that because I couldn't find it on TV, finding a handful of apps on P2P wasn't necessarily that bad. Afterall, I
Get Microsoft Sharepoint Portal Server (Score:2, Insightful)
It won't be a solution, because after a while people will get used to having the network and being able to find the documents on other people's machines, and they will ask for text searching, version control, and so on. Not sure what Microsoft wants for the Sharepoint and what are the Linux alternatives, but it seems to be worth it in your case.
Why not cron FTP? (Score:4, Interesting)
Re:Why not cron FTP? (Score:3, Interesting)
Besides, once you had the network set up, I'm sure other uses for it would manifest themselves.
Re:Why not cron FTP? (Score:3, Informative)
rsync, anyone?
A local mirror of the files from the public ftp server definitely sounds like the way to go. Just make sure everyone in your office knows to grab stuff from the local (ftp|nfs|samba) server. This will provide faster access for everyone in your office, and reduce load on the public server.
P2P is a solution for when you dont have control over the infrastructure to set up somethin
Why do you want to make things more complex? (Score:5, Insightful)
What about Nullsoft WASTE? (Score:4, Insightful)
Bittorrent (Score:5, Informative)
The source is available, too.
Bittorrent (Score:4, Informative)
you've got it backwards (Score:4, Insightful)
If you're a small garage band trying to advertize yourself, there's nothing wrong with throwing mp3s of your performances on kazaa. Anything else that you created yourself is legitimate, too. Same with uncopyrighted works (like the complete works of Shakespeare, for example).
The only real problem with file sharing is that nobody wants that stuff, they all want the copyrighted stuff
Oh, and I downloaded Mandrake, RedHat, and Knoppix ISOs from BitTorrent. Those were totally legit uses.
Re:you've got it backwards (Score:2, Insightful)
I believe some band will make it big this way someday and from that point other bands and artists in general will move towards p2p distribution.
Re:you've got it backwards (Score:2)
I'm not so sure about that -- if you don't know the name of the band to search for, it's pretty hard to find their stuff, especially if you're not looking for it. Maybe it's great and you just don't know? They need some better advertising, at least
Re:you've got it backwards (Score:1)
Or maybe you've got it backwards and it's just that all the stuff they want is copyrighted.
Songwriters (Score:1)
If you're a small garage band trying to advertize yourself, there's nothing wrong with throwing mp3s of your performances on kazaa.
What about the copyrights owned by the songwriters?
Re:you've got it backwards (Score:2, Insightful)
Semi Relevant Article (Score:4, Informative)
Anyway, it's entirely possible to legally distribute content via p2p, I'm amazed that nobody has really turned it into a fully fledged service.
Video Game Application (Score:3, Interesting)
Hopefully this thing works well and will spur game developers to support the concept (winkwink). I fully anticipate games to ship with in-game P2P content delivery systems in the future. Integrating that with chatrooms and game lobbies is the next logical progression. Share some levels or models while you chat it up. Release new mods through P2P and stage a chat release party, all within the game architecture. Plus, the user base is already trained in the software.
Re:Video Game Application (Score:2, Insightful)
Re:Video Game Application (Score:2)
Grendel (Score:4, Informative)
All you need is to imagine a Beowulf cluster of...
Waitaminute!
You actually could think of this as a Beowulf cluster! The main twist is that each node in the network is being used interactively, rather than just acting as a slave that churns away on data chunks autonomously.
You don't state what kind of systems your colleagues are using, but if you're using Macs, then Rendezvous mDNS networking can take care of the "plumbing" part of the problem for you -- everyone can instantly start publishing their shared resources, and the trick then is to just figure out a way to search who has what content.
The search function could be done from a machine set up to automatically spider everyone's content & basically set up a little in-house search engine, with links back to each user's version of "http://johndoe.local/weather/data/2003/08/21/1530 _nws" or whatever.
If you're not running Macs, well that's a problem on several levels :), but the mDNS spec is an open standard, and it is IIRC available as an Apache module. There's mod_rendezvous [mac.com], but it seems to have stalled with an OSX version only -- porting to Linux shouldn't be bad but is left as an exercise for the reader. There also seems to be the Net::MDNS::Server [cpan.org] & Net::MDNS::Client [cpan.org] Perl modules on CPAN, but they seem to have been born & stalled in the same week back in June. Not sure what that means.
In any case, if you can set up a spontaneous mDNS network, then that would solve the problem of getting every node on your network to be able to advertise what resources are available to other nodes on the network. The step after that is to set up a search interface, and that's really a solved problem -- any Perl hacker comfortable with LWP should be able to whip up a reasonably good search mechanism using &/or extending existing tools.
If you manage to get this to work, it would be interesting to read a writeup of how the lego parts end up being assembled :-)
Re:Grendel (Score:1)
Waste or Konspire (Score:2, Informative)
or Konspire
'open source' like music label thats pro-yro (Score:1, Offtopic)
Here's a link to download their first release, WTF! The Madonna Remix comp in Ogg, MP3, or AAC... [justablip.co.uk]
also check out the ARTICLES on the site, I think people on here would be into it.
ant
--
))
((
c[_] bLiP
www.justablip.co.uk
Re:'open source' like music label thats pro-yro (Score:1)
Madonna Remix
Did you get permission from the songwriters?
Try these P2P protocols (Score:3, Informative)
JXTA [jxta.org]
BitTorrent [bitconjurer.org]
Or, you can create your own Gnutella client, using some an open-source Gnutella package, like JTella [sourceforge.net]
no, not BitTorrent (Score:3, Informative)
Additionally using BT would turn out to be -more- wasteful, for two reasons: One, because to make the data available you'd have to automatically retrieve it from the origin server, regardless of if there is a demand for it or not. Secondly, BT is still a somewhat of a niche protocol and so there's a good chance that there would be people that say "screw this, I don't want to instll Python and wxWindows just to get this file that I can download with Mozilla in about 3 seconds."
I'm with the person that said setup a caching proxy server. Squid will do this perfectly, and it doesn't involve making the users change at all -- it's all behind the scenes if you set it up as a transparent proxy. There will be no wasteful cron-job downloading since Squid will simply cache whatever the users are requesting -- if no one needs data for some period there's no point in wasting the weather site's bandwidth on some cron job ftp thing.
Please don't be so fast to suggest something like BitTorrent just because it's trendy.
Re:no, not BitTorrent (Score:2)
Agreed. It's an outstanding concept, but it's both bitstorm-oriented and relies on goodwill: it assumes many downloaders, who then make the decision to keep their download window open for an indeterminate time. It's an outstanding social protocol, but a poor forced dissemination protocol.
Copyrights??? (Score:2, Interesting)
Let's say Site A provides the data on a non-commercial basis freely to users but charges for commercial use. They require you to go through an agreement in order to get this data for personal use or else sign up and get a license for commercial use... Site B can't just go and get this data from Site A and procede to share it out. Because those who got it from Site B would not have clicked through t
example of legitimate P2P platform and apps... (Score:2, Informative)
NullSoft Waste (Score:2)
Transparent Proxy / Cache server (Score:2)
Of course you could use squid for this but, there are also several commercial products that do very well.
The most important thing to remember is to make sure the server is big and powerful. The server must have at least one fast processor, oodles of memory(1 gig or more) and it needs to have a very fast disk subsystem. I recommend hard
uhhhh caching proxy (Score:3, Insightful)
Then go get a drink, for not having to re-invent the wheel, or cost your company much ( if any ) money.
First thought was local mirror... then... (Score:3, Insightful)
Ask the public FTP provider to put up Torrent files, or equivalent, right on the FTP site, or in a README. Alternatively, if they aren't interested or are too busy to set it up themselves, ask them to put a description and link to a webpage that YOU control in a readme file. Something along the lines of, "Our servers are busy. A good samaritan maintains an alternate download method at this URL. You may get faster results from there." Perhaps toss in some MD5 hashes so that people can verify data if they so desire, and then hope for the best. With any luck, your company wins, other groups win, and the provider wins.
Pretty much any P2P network would work, but you would probably have more support from IT groups for BitTorrent than eMule/eDonkey or Gnucleus.
JXTA Works and yes, P2P is Legit (Score:1)
Yes there is a legit use for a lot of P2P. I work at Verizon. I have also worked for Boeing and have a couple open source and one private venture. All use P2P in 100% legitimate applications and only one is sharing files.
Use JXTA. It is mature and far better than Gnutella because of the level of control and diversity. Gnutella just shares files and can't really improve the efficiency of distribution.
Head to jxta.org to start.