Guaranteed Transmission Protocols For Windows? 536
Michael writes "Part of our business at my work involves transferring mission critical files across a 2 mbit microwave connection, into a government-run telecommunications center with a very dodgy internal network and then finally to our own server inside the center. The computers at both ends run Windows. What sort of protocols or tools are available to me that will guarantee to get the data transferred across better than a straight Windows file system copy? Since before I started working here, they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals."
Robocopy? (Score:5, Insightful)
Robocopy? http://technet.microsoft.com/en-us/magazine/2006.11.utilityspotlight.aspx [microsoft.com]
W
domyjobforme tag (Score:2, Insightful)
I love it! Haha... that's probably one of the better tags I've seen.
BitTorrent (Score:5, Insightful)
rsync should do the trick (Score:5, Insightful)
hi there,
why don't you get cygwin on both the systems and then do a rsync ?
between your own network, you might want to use robocopy(http://en.wikipedia.org/wiki/Robocopy).
BR,
~A
Line endings! (Score:5, Insightful)
they've been using FTP to upload the files, but many times the copied files are a few kilobytes smaller than the originals
Twenty bucks says you're converting from Windows line endings (/n/r) to Linux line endings (/n).
Use binary mode and you'll be fine.
Re:TCP? (Score:5, Insightful)
TCP has timeouts. The FTP client and server probably have timeouts. Eventually, some bit of the system will decide the operation is taking too long and give up. The FTP client is probably reporting an error, but if it's driven by a poor script no-one will know.
Re:TCP? (Score:4, Insightful)
I bet it is file systems with different block sizes rounding slightly differently, and an OP that does not understand.
Re:TCP? (Score:3, Insightful)
FTP, while in ASCII mode, can try to translate line endings. If the carriage returns were removed, in order to be UNIX compatible, the file size would have been reduced.
Most FTP clients allow the enabling of a binary mode which prevents the conversion from happening.
Re:Robocopy? (Score:4, Insightful)
It might be using Windows copy protocols, but it definitely is not like copy/paste. It's restartable for instance. It's way more reliable.
We have to copy large files to our office in China. FTP always fails. Windows copy via Explorer often fails, but it is also incredibly painful to do when latency is high and one is browsing over the network. Robocopy (depending on system setup) will motor through and is very persistent when there's a connection hiccup. You definitely want restartability if you copy large files are a couple of hundred MB an hour.
I'd say make sure to break the files up in to chunks if they're large. Also, run 2-4 robocopies in parallel if the latency is high as this will give better throughput. It can do funny things to Windows though (maybe other things wait on some network handle and seem to freeze until one of the robocopy processes moves on to the next file).
Also, consider doing it over a Cisco VPN. It seems to add some robustness if there is packet loss. I often had trouble access servers in the US when I was living in China due to packet loss, but no such problem over a VPN (zero packet loss, but very slow instead, which is better).
Re:domyjobforme tag (Score:3, Insightful)
Too easily thrown around if you ask me. He's not looking for anyone to set it up, he just wants some options. Isn't that what community is about?
Re:UDP. (Score:1, Insightful)
You're kidding, aren't you?? (Score:5, Insightful)
You are kidding about this, aren't you?
Let me get the facts straight:
- you have "mission critical files", and the network you're transferring them over is so incredibly badly managed that it doesn't support reliable data transfer
- you want a technical workaround for this brokenness.
If this is the case, you don't have a technical problem on your hands; you have a political one.
"Mission critical" has a meaning: it means critical to the success of the operation. I.e. without these files, your operation or someone else's operation will fail.
If your management believes that your files are "mission critical", and you're facing a problem of this sort, you need to document the difficulties you're having, along with measurements to support your claims and then make a clear statement that as long as your network path is completely broken, you are absolving yourself of responsiblility for the correct transmission of these files.
If your management doesn't do anything about this, then the files are not "mission critical".
Re:TCP? (Score:3, Insightful)
Re:TCP? (Score:3, Insightful)
Binary verses text mode?
Lousy windows file system screwing up on one or the other end.
Sparse files.
Windows "fixing" the data during transmission.
Loss of packets, and no error checking.
Windows.
Re:You're kidding, aren't you?? (Score:1, Insightful)
Well, the original post never said whose mission. It could be that the employer with the "mission critical" need does not control the network or the computer choices. So, they charged the original poster with fulfilling the mission in spite of the given handicaps.
Now anything that gets my employer paid is "mission critical." I may be (and often am) at the mercy of my customers equipment. But my job is to work through or around those obstacles. Because removing them properly is not an option.
Re:You're kidding, aren't you?? (Score:4, Insightful)
SSHFS (Score:3, Insightful)
I use sshfs file mounts for all office document file sharing and such, not just one time transfers. SSH encryption security, with the ability to open and edit files over the network. No goofing around with samba or windows file sharing. Regardless, some sort of ssh or sftp at least.
Not sure about getting it to work on windows, but there should be some options.
Re:Any encrypted transmission protocol actually (Score:5, Insightful)
Re:You're kidding, aren't you?? (Score:3, Insightful)
I think you missed the part about the government being involved
Z-Modem FTW! (Score:5, Insightful)
Crappy connection? Resumable transfers? Slow connections? Sounds like the good old BBS days!
Z-modem is your answer.
Re:You're kidding, aren't you?? (Score:1, Insightful)
I think this is a technical problem unless you are working using COVER YOUR ASS engineering.
You can start working under the assumption that a new line is cost prohibitive or impossible, (more expensive than a week of salary for a engineer, or maybe they are replacing the OS of the voyager 2). Also the problem sounds very simple so i don't understand why to make that much of a hassle.
Remember you are paid to solve problems, not to bitch about them.
Re:You're kidding, aren't you?? (Score:1, Insightful)
Hmmm.... Here's a thought? What if the "Mission" is something related to a some sort of military activity and the data that is "Critical" is held in a location with spotty connectivity and severely constrained bandwidth (say a beach position with a little satellite dish). I am sure your hand washing memo will be very well received by all those folks who are put in harms way because of your willingness to make it someone else's problem.
You simply cannot assume away the problem or try and fit the problem to meet your expectations. Microsoft and IBM have made a reputation on trying this approach, but reality has this nasty habit of ultimately dictating the terms of the problem in spite of whatever massive resources are expended to the contrary.
Technical work arounds are the nature of the beast in the "real world". Perhaps a dose of exposure there will give you a little more empathy and perspective from which to approach these types of issues in the future.
I don't feel like registering at the moment, so I guess I will go with Anonymous Coward for now... (Tom for short) :')
Re:UDP. (Score:3, Insightful)
TCP does this pretty well on 99% of the internet *and* the internet is aware of TCP. Its only very "different" connections that things can make a real difference. AKA the microwave link. Though we have wrapped the link with specific hardware/software layer that lets IP work well over it in our cases.
Also when people "roll" their own "superior" UDP transfer protocol, many don't bother to check why TCP does what it does. Flow control is *needed* with ACKs and resends --well any connection, buffers are not infinitely big. There is the 2 generals problem etc. Its not as strait forward as many think.