Ask Slashdot: Do You Move Legal Data With Torrents? 302
An anonymous reader writes "We've recently seen a number of interesting projects come from bittorrent.com, including Sync and SoShare. I sometimes use torrents to move several GB of data, especially when pushing large bundles to multiple destinations. It's mostly a hodgepodge of open source tools, though. Apart from anecdotes and info from bittorrent.com, details are thin on the ground (e.g. the Blizzard Downloader). I have two questions for the Slashdot community. 1) Do you use BitTorrent to move data? If so, how? i.e. What kind of data and what's the implementation? 2) If you've looked at torrent clients/tools, what's missing in the open source ecosystem that would make it more useful for moving around large blobs of data?"
I use it for linux distributions (Score:5, Interesting)
The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.
Re:I use it for linux distributions (Score:5, Interesting)
While working at Fermilab on the LHC CMS data taking team, I used bittorrent to speed up re-installs of thousands of worker nodes. I was able to saturate 10Gb Ethernet links this way, and could reinstall ~5500 Linux boxes within 10-15 minutes (with only two initial OS source servers).
Yes, Bittorrent is not just for piracy.
Re:I use it for linux distributions (Score:5, Funny)
I nearly jizzed in my pants reading this. Nerdgasm.
Re: (Score:3)
Hey, I practice safe sex. I have a firewall and every porn file gets scanned first before use.
Re: (Score:2)
Re:I use it for linux distributions (Score:4, Informative)
How would you use broadcast or multicast to distribute an OS? Call me ignorant, but how would you do that in practice?
Pretty easy to setup using Windows Deployment Services
http://www.windows-noob.com/forums/index.php?/topic/452-how-can-i-multicast-an-image-in-windows-deployment-services-windows-server-2008/ [windows-noob.com]
Or if Linux is your preference
http://www.udpcast.linux.lu/ [linux.lu]
Or Clonezilla has a multicast restore function
http://clonezilla.org/clonezilla-SE/use_clonezilla_live_in_drbl.php [clonezilla.org]
Re: (Score:2)
You don't multicast images to Linux machines when you're using configuration management tools. PXEBoot->Bare Image Install->Puppet system configuration upon first boot based on machine grouping/criteria.
Disclaimer: I'm OP.
Re: (Score:2)
Here's how Norton did it, back in the late 1990's: http://www.symantec.com/business/support/index?page=content&id=TECH106806 [symantec.com]
I recall using this in college, in 2003, to reimage our 'learning' workstations. (After we'd break them, like discovering that Windows 98 SE would let you format the OS volume, and not crash.)
Re: (Score:2)
Use multicast for this.
Stop using shitty networking gear that squashes multicast traffic. Use erasure or fountain coding to transmit your datagrams so clients can compute ones they missed. Have clients individually request any remaining lost datagrams over unicast.
Torrents are a very specific solution to a very limited problem in networking; specifically, people being too cheap to invest in proper co-located servers for on-demand traffic. Stop trying to apply it everywhere else.
Re: (Score:3)
That's not the entire point.
If you want to save bandwidth and still distribute your data, then crowdsource your downloads with bit torrent.
Linux, MMO games, game mods, etc. All excellent uses of bit torrent.
Re: (Score:2)
The entire point of swarm topology is to move data to a lot of places at the same time. If you just need to get data from A to B without sharing it with anyone else, rsync it.
One huge advantage of bittorrent is that error checking/correcting is built in.
Although you would still need to re-download the block with the error, it's a very small amount of data (usually a few KB). This solves the problem of errors that happen after any data transport verification, as most bittorrent clients can be configured to do a final re-check after the torrent is complete. In addition, the data transport can be encrypted if you need it to be. Although it's not the strongest of encryptions, it
Re: (Score:3, Interesting)
Yup, it is one of the few ways that darn near any user out there can contribute back to Open Source.
I can code, some, but not very experienced with c/c++ - just haven't had the need to do it. I hate writing documentation. I do file bug reports, but the stuff I use is pretty darn stable.
So, to give back, I seed iso images for 24 hours or 100gb in upload of any Linux distro release notice I see here on slashdot, even if I don't use that distro.
Re:I use it for linux distributions (Score:5, Interesting)
To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).
The advantage I see rsync having is it is useful for automated (backups) of a large collection of files vs gzipping and copying via scp.
Although git also does a great job of that with concurrent revisioning built in.
It all boils back down to using the right tool for the job.
Re: (Score:2)
It all boils back down to using the right tool for the job.
The wise keep as many tools in their tool box as possible. You seem to be among the wise.
rsync transfers changed PARTS of files (Score:2)
To be fair, rsync + ssh is equally secure as scp. I actually think rsync uses scp in that situation (please correct me if I am wrong).
rsync and scp operate somewhat similarly if the file does't exist all on the destination. If an earlier version of the file exists, rsync transfers only the changes. Therefore it couldn't use scp - rsync does things that aren't possible with scp.
Since sometimes you have to use rsync because scp can't do it and I don't care to memorize redundant options, I normally use rsync even with scp would do.
Re: (Score:3)
vs gzipping and copying via scp
"scp -C" will compress as it transfer, no need to gzip.
Re: (Score:3)
...and if the connection breaks or anything happens, you'll have to restart copying everything from the beginning.
Interestingly, rsync can even resume a transfer that was started by scp but then interrupted - neat!
Move my legal data? (Score:5, Funny)
I let my Lawyers handle that. It's what they're paid for, isn't it?
Re: (Score:2)
Speaking as someone who, at any moment, only has to hit up arrow a couple times to see an "rsync .. -e ssh ..." line, I'm telling ya: it's all the same.
Re:I use it for linux distributions (Score:5, Insightful)
$ env |grep RSYNC
RSYNC_RSH=ssh
Worth putting right in /etc/profile so anyone who doesn't want it can disable it if they want.
It is an entirely sane default.
Re: (Score:3)
Not by default, though. You have to use a flag.
Re: (Score:3)
Re: (Score:2)
Probably the fastest way to get your latest linux distro (and Libre Office).
Absolutely (Score:5, Funny)
All of my Torrents are legal data. What else would I use Bittorrent for besides Linux distros and Humble Bundle games?
Re: (Score:3)
Amateur electronica, generated in Garageband?
Re: (Score:2)
Eclipse, of course. Although I'm unsure whether the submitter meant "moving data" as using bittorrent for file transfer, not downloading.
Re: (Score:2)
Distributing home made erotica of course!
Re: (Score:2)
Porn?
No - Resources (Score:3)
Moving large data requires resources. In the case, of bittorent most things don't qualify because it's a distributed network, if 10 people in the office have the file and all know how t seed / use bittorent, you'd still be throttled by your bandwidth. Bittorent has however time and time again shown that it's distributed architecture can get something out to the masses very effectively.
Re: (Score:2)
I don't see why Bittorrent wouldn't work for an office of 10 people; its strength is copying data over distributed networks. For an office with 10 people, assuming they're on wired Ethernet links rather than WiFi, all 10 of those links will be connecting to one or more switches, which are able to handle full 100Mb or 1Gb speeds, duplex, between the computer and the switch, simultaneously. Using BT to copy the data might be a little slower than just using scp if it were only one PC, but the total transfer
Re: (Score:3)
If the 10 people are in the same collision domain, torrenting won't make things faster. Actually it can slow things down with unneccesary collisions.
Re:No - Resources (Score:5, Insightful)
Serious question... How long have you been using a network HUB instead of a network SWITCH?
Re: (Score:3)
Probably the same guys that never upgrade from their 486DX120 because it's fast enough to run Lynx.
Re: (Score:3)
Collisions? Are switches that expensive?
Re: (Score:2, Informative)
You're confusing broadcast domain with collision domain.
On a switch, a collision domain is limited to the link itself, ie, collisions aren't possible. However, all ports on the switch (or all ports in the same VLAN, if the switch is manageable) share the same broadcast domain, which merely allows for all ports to contribute to saturation, but there still won't be any collisions
Re: (Score:2)
Nope. (Score:3)
I mean, other than the Blizzard stuff, no, I don't use bittorrent at all unless I'm downloading movies (usually) or software (sometimes).
Rapidshare/megaupload/etc work much better for my one-off transfer needs, while I leave media distribution to the masses via Youtube, Vimeo, Bandcamp, and media collaboration to Dropbox and sneaker-usbdrive-net (especially for big projects).
Yes, I have (Score:5, Interesting)
In a word? YES! (Score:4, Informative)
I specifically do not torrent anything that has copyright issues but I do seed a number of Linux distributions and development tools which do not prevent distribution in their licenses. Downloading anything using torrent is effectively distribution of the material too, so you had better KNOW that the license allows you to make copies and give them away.
You folks that torrent movies and stuff that is not in the public domain are crazy in my book.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Very, very bad analogy as the concept of copyright is a good thing, the concept of racism is not.
Remember that only thanks to the existence of copyright the GPL and all those other FOSS licenses can exist. Without copyright it'd all be public domain.
It is the current implementation of copyright laws (primarily: too long terms) and all the other laws that surround it (e.g. DMCA blocking reverse engineering and circumvention of DRM) that is the problem.
Move Legal Data With Torrents? (Score:5, Funny)
Only a complete fucking moron would move legal data with torrents.
A lawyer is obligated to preserve his clients' confidences. When you store your information on somebody elses server or servers you are giving up custody and control over some of those confidences. In that situation you are entirely dependent upon the strength of your encryption.
That encryption might be good today or tomorrow, but how good will it be five years from now or ten years from now when quantum computing or the next best thing becomes available for codebreakers.
Don't risk a lawsuit from a pissed client!
Re: (Score:2)
In this context, I believe legal meant "not illegal". Or "data I own" or "data that nobody will sue my ass off for moving them around"
Re: (Score:2)
Aren't you worried about getting the deaf penalty for using their network racehorses to trade TV shows containing sax and violins?
Oh.
Never mind. [wikipedia.org]
Re: (Score:2)
What are they gonna do? Fire a machine gun by your ears a few minutes until you can't hear any more as a sentence? Yeah, that would have the potential to make you go deaf...
Can you use it to move a 300 megabyte hosts file? (Score:5, Funny)
Re: (Score:2)
Linux ISO's mostly (Score:5, Informative)
At work I need to install several different types/versions of linux OS's for testing. I always torrent the ISO as a way of "paying" for the image that I'm using.
A few years back, we did some experimenting with torrents over the Teragrid 10GBe backbone, to see how well that worked over the long haul between IL and CA. With just 2 endpoints, even on GBe, it wasn't better than a simple rsync. We did some small scale test with less than 10 cluster nodes on one side, but still not as useful as a Wide Area filesystem we were testing against. Bittorrent protocols just aren't optimized for a few nodes with a fat pipe between them.
I am interested in looking at the new Bitorrent Sync client to see how thanks for our setup. We have many users with 10's of TB's of data to push around on a weekly basis.
+1 Linux distros. Only for multiple recievers (Score:4, Insightful)
with just two end points, it wasn't better
For point-to-point transfer to large amounts of data, the protocol does't matter, as long as the protocol is sane. The time spent moving data bytes will be much higher than any protocol overhead. rsync is roughly optimal because it won't transfer portions of the file that the receiver already has. BitTorrent is for distributing data to many destinations.
Re: (Score:2)
Even at home I noticed that copying large files over wireless via SMB is much slower than copying them over an SSHFS mount (getting previously unseen transfer rates, actually). However, the SMB mount is more responsive when exploring files.
In my experience, protocol can matter a lot.
Re: (Score:2)
1 RTT per GB? For a Maildir mail spool, yes (Score:2)
Re: (Score:2)
BitTorrent is, conceptually, best used as a reimplementation of multicast. Multicast is probably far more efficient when it comes to the actual data distribution, but multicast (specifically, routing multicast) is one of those blackbox things that not a huge number of people understand. Last time I checked, I couldn't route a multicast source from a Comcast connection and have the data arrive on a receiver on the Cox network.
However, there was still a need for a protocol that could effectively do one to man
Re: (Score:2)
I agree: torrent can't really saturate a 10GE
Point-to-point torrents can't saturate a 10GE. Get enough nodes involved and they can.
Re: (Score:2)
Yep- Linux (Score:3)
Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).
Re: (Score:3)
Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).
For such tiny things (latest UBCD is 486MB), I pretty much seed forever, as it doesn't really cost me anything. I'm at 67:1 on the latest UBCD, and over 400:1 on some other much smaller torrents.
I leave them running because I don't artificially limit my upload rate on a per-torrent basis, only as a total for all torrents (and that limit is quite high, as I have Verizon FiOS). It really bugs me when I try to download an older torrent and the only seeds are uploading at a few KB/sec. Even on a 100MB torren
Re: (Score:2)
Re: (Score:2)
Just about the only time I use torrents is when downloading Linux distributions- Mageia, Fedora, CentOS, etc. Occasionally iso's for grub magic, ultimate boot CD, and such. All of that legal. And I usually leave it up at least long enough that my share ratio is 100% (1.0).
Even better is if you aim for a share ratio of 2.0. With a share ratio of 1.0, you only "give back what you take", so the swarm stays as strong as it was. That's good. But sending back an "extra copy" gives your contribution to make the swarm stronger.
occasionally (Score:4, Informative)
Some (Score:3)
Linux distros, free movies, free games...
I tried to switch to Deluge but it couldn't handle a file with a Japanese character in its name...other than that, only things that I think many torrent clients could use is the ability to accept magnet downloads through a drop folder somehow, and searching & better sorting/filtering options for downloaded torrents.
We deploy VM images to our developers over BT (Score:2)
gittorrent (Score:5, Interesting)
the one thing that would help enormously would be to have git be *truly* peer-to-peer distributed. not "yeah shure mate you can always git pull and git push, that's distributed, and you're a peer, right, so... so... git is peer-to-peer and distributed, so what are you talking about you moron??" but "at the network level, git pull and git push have a URL type that is **TRULY** peer-to-peer distributed. to illustrate what i mean, i would like to be able to do the following - with all that it implies:
git clone magnet://abcdefg0123456789/gittorrent.git
if you're familiar with magnet links, you'll know that there is *no* central location: a DHT lookup is used to find the peers.
now, what wasn't clear to the people on the git mailing list when i last looked at this, was that it is possible to use bittorrent to do git pack objects, by creating a file named after the pack object itself. and what wasn't clear to sam (the last person who tried to put git over bittorrent) was that you *MUST NOT* make use of bittorrent's "multiple file in a torrent" feature, because bittorrent divides up its data into equal-sized blocks that *do not* line up with the files that are in them, which is why when you download one file in a torrent you almost always end up with the end of its preceding file and the start of the one after it, as well.
the idea i came up with is that you create *multiple* torrents - one per git object (or git pack object). if you want to pull a tree, you create a torrent containing *one file* which is the list of objects in that tree; gittorrent would then know to map each of those objects onto yet *another* torrent (one per object), repeat until all downloading happily. gittorrent objects are of course named after the hash, so you can pretty much guarantee they'll be unique.
and, adding in a DHT (a la magnet links), you are now no longer critically dependent on something like e.g. github, or in fact any server at all.
to answer your question in a non-technical way, mr anonymous, i think you can see that i feel it would be much more useful to have development tools that use bittorrent-like protocols to share files-as-revision-controlled-data (and, if you've seen what joey hess is doing with bittorrent you'll know that that's a hell of a lot - including storing home directories in git and doing automatic distributed backups)
Re:gittorrent (Score:5)
Intriguing idea, but I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over. The problem with using bittorrent for this is distributing NEW data. If the protocol could cope with a seed appending data to the torrent without having to create a whole new .torrent file, then this could be awesome. As it is, you're just changing the problem from "how do I send out new versions of files when I commit something" to "how do I send out new versions of the .torrent files every time I commit something"
Re: (Score:2)
I tried subscribing to your newsletter and I keep getting the Sept. 24, 1998 edition over and over.
A new meme is born!
Re: (Score:3)
I see your flamebait already fooled some mods. However, you're supposed to attack the idea, not the man. And you failed spectacularly at that.
The key component in what he's proposing would be the DHT; not even necessarily the same DHT as used by standard BitTorrent clients. The use (or not) of the BitTorrent specification and its intricacies would be just an implementation detail.
You're free to be one of the "can't be done" people; it would be wise not to advertise it too much around here however; as it's n
Damn right (Score:2)
Linux ISOs
VM Images
Backup Images
Home Movies
etc...
Re:Damn right (Score:4, Funny)
Game download (Score:3)
I got a game called War Thunder via bit torrent. What you do is download a small installer program from the publisher's website and run that. The installer automatically connects to BT seeds and peers and downloads the actual game itself.
There is no other way to get War Thunder. I suppose since they're a small publisher, their web server can't handle distributing the 13 GB game file to tens of thousands of users.
Re: (Score:2)
I hate it when developers do this. Offer a torrent as your default download, sure, but put a direct download buried somewhere in your support pages.
Some people (like me) have net connections that just crap out on torrents. I can download a large file reasonably well, but the same file in a torrent will take weeks.
And some people (a majority, in some nations) have caps, and a torrent-based downloader eats into that quite a bit.
Multi site sync (Score:3)
I have a fairly large and growing (3.7TB) dataset that needs to be replicated nightly to a bunch of different sites. By the nature of the dataset nothing is ever removed or changed just new files added. It needs to be copied out to a half dozen locations that have as much outbound bandwidth as the primary. So a cron job sets up the torrent every night and all the remote sites pull data from the primary and reshuffle it between themselves.
Re: (Score:2)
I may miss something, but isn't this the ultimate job for rsync, to copy only the new bits? Or do you take a diff and distribute only that?
NO (Score:3)
Bit torrent is a good data -distribution- tool, not a data -mover-, and it would be lousy to play that role. There are at least a dozen possible open solutions for moving data from point to point, but I have no idea why you'd use a protocol/tool stack that are designed for broadcast/graph distribution to do so.
An off the top list:
1. NFS
2. SMB
3. FTP
4. SFTP/SCP/rsync
5. HTTP/HTTPS
6. sz/rz
7. iscsi
8. DFS
9. AFS
10. UFTP/XFTP
The question should really be what exactly do you see ad being deficient about all these protocols that deems it necessary to re-invent the wheel yet again?
What does THAT matter? (Score:3)
Punish the technology because of how it is used? I thought we grew past that notion already.
At some point, one of the few remaining ways to get good information and news will be through these rougue channels and methods. Do we have to keep re-hashing the same ridiculous notions? How about we ban types of music based on the fact that thugs and criminals like it and glorify killing?
Re: (Score:2)
I thought we grew past that notion already.
Where have you been?
Guns, drugs, ... (Score:2)
Punish the technology because of how it is used? I thought we grew past that notion already.
If we had you'd be able to buy any weapon or drug you want without government interference or oversight.
The NRA could go back to its original functions of training and research, and the FDA and DEA to could be replaced by Underwriters' Laboratories and Consumers' Reports.
As you can see we have a long way to go.
Facebook Does (Score:4, Interesting)
The big advantage (Score:3)
The big advantage of using BitTorrent over many other protocols for moving large amounts of data (as opposed to distributing it) is reliability - or lack of it. When you're moving large datasets, you don't want it to crap out and not be able to resume 90% in.
Sure - rsync has the ability to resume, but it requires explicit command line options. It's a terrible feeling to realise you just restarted a 10GB+ transfer instead of resuming it.
SOLR at Etsy.com? (Score:3, Interesting)
The folks at Etsy do it to replicate SOLR:
http://codeascraft.etsy.com/2012/01/23/solr-bittorrent-index-replication/
Not sure if that's what you mean.
Red vs Blue (Score:2)
Back in the pre-YouTube days, Rooster Teeth distributed their videos using Bit Torrent to relieve their own HTTP load. I think they gave BT users the incentive of downloading earlier to encourage its use.
All data is legal (Score:2)
no (Score:2)
I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.
About the only thing I could think of is a linux distro, but again the only thing I am bound to cobble together is a lightweight debian mix for maintance / repair / recovery when someone brings me their computer to be fixed and winders is all trashed.
Re: (Score:2)
I dont have anything worthy of having to mass source it, and my 3$ a month "unlimited bandwidth" website has taken 400 gigs in a month downloads before without a sweat.
That's an awesome price, even if the overall speed isn't that great.
Everything I have found that gives you more than a few GB of disk space and truly unlimited bandwidth costs a lot more than that per month.
Not really a fan of it. (Score:2)
It's a good idea for sharing, but I'm not a fan of the way it loves to hog the Internet connection, it starts connections out the ass, flooding the network and just slows everything down. I tend to use wget for pretty much everything; with a decent server, it gets by just fine with only one connection. I also like the idea of maintaining at least somewhat-accurate timestamp data whenever possible, and BitTorrent doesn't seem to have a concept of that. And also, I like to maintain my own checksum files, s
Re: (Score:2)
If you're using a bittorrent client that doesnt' preallocation the file, then you should switch to a non-shitty one that preallocations, avoiding the entire fragmentation problem completely.
Or you could use a filesystem that didnt' suck.
Re: (Score:3)
You know you can limit the number of peers you connect to as well?
E.g. I have it set to a maximum of ten torrents active at any one time. Each torrent can have no more than 60 peers. So, total, 600 peers maximum at any one time. Except, I also have it set so that I do not connect to more than 275 peers, ever. So that 600 number, irrelevant.
I.e. these are all problems that, well, aren't.
Moving data with torrents (Score:2)
Only Legal torrents (Score:2)
I use Torrent for many legal uses :
- Blizzard downloader (WoW, Starcraft2, Diablo 3)
- Humble Bundle
- Linux Distributions
But also, from time to time, some free tools, some F2P games that use BitTorrent for distribution, ...
Movies and TV programs. (Score:2)
Yup, lots, pretty much always legal. (Score:2)
I pretty much always use torrents to download OS installers (Linux, etc.). I don't know whether I've ever downloaded anything that wasn't totally legal. Probably have at some point, but I mostly use torrents for things where I want to be contributing bandwidth back to the community.
Yes: Linux distros (Score:2)
Wait... (Score:2)
Wait, you can move illegal data using torrents? Who would have thunk!?!
Haven't used Torrents in years. Many years.
yes I do. (Score:2)
bit torrent is great, because everyone with consumer broadband becomes and instant mirror when they download it.
Microsoft ISO's (Score:2)
We use bittorrent to move MS ISOs at work. We are an Educational institution and as such we are allowed to offer students access to MS products. (not the Office suite, but OS's .NET and the like) This is done via the Dreamspark site where students can get their license keys but from our internal network they can use bittorrent to get the ISOs
So we legally give access to MS OS via torrents (^_^)
Re: (Score:3)
Does this seem like fishing expeditions by patent trolls?
No need to go fishing here.. Just find the torrent tracker and connect... Volia, you have a list of everybody who is distributing the material by IP address. It's all about tracing down the IP's and sending out the collection letters and cashing the checks from there, assuming you actually have the permission of the copyright holder to do so...
Re:Patent Trolls? (Score:5, Funny)
i tend to run through a Romanian based VPN almost exclusively for work(my own personal. I wouldn't trust an outside vps with corporate information) related activity, and i can absolutely confirm what he is saying. I've been on the phone with my vpn provider before(hes an old personal friend), and he says "OH BOY MAIL TIME, now i get to have more firewood from your american lawyers" as to which we both chuckle.
Re: (Score:2)
Re: (Score:2)
If I remember correctly, some online games use P2P to distribute updates legally (though they might not use bittorrent).
Before I moved to Steam, torrents were the *only* way I could get updates for Company of Heros.
Re: (Score:2)
'custom software' ... right because there aren't any libraries for multicast file distribution already or anything.
You fail at the Internet. Multicast file distribution is probably older than you are.