FTP: Better Than HTTP, Or Obsolete? 1093
An anonymous reader asks "Looking to serve files for downloading (typically 1MB-6MB), I'm confused about whether I should provide an FTP server instead of / as well as HTTP. According to a rapid Google search, the experts say 1) HTTP is slower and less reliable than FTP and 2) HTTP is amateur and will make you look a wimp. But a) FTP is full of security holes. and b) FTP is a crumbling legacy protocol and will make you look a dinosaur. Surely some contradiction... Should I make the effort to implement FTP or take desperate steps to avoid it?"
hmm (Score:5, Interesting)
Use sshd (Score:1, Interesting)
Ftp is a major pain for firewalls because unless you use it in passive mode it opens another port back after the initial connection.
HTTP is faster (Score:1, Interesting)
Why isn't there SFTP support from the browser? (Score:1, Interesting)
SCP (Score:5, Interesting)
It all depends on the application. I only use SCP to move files around if I have the choice, just because I like better security if I can have it.
But if you want to offer files to the public, I'd recommend offering both FTP and HTTP so people can use the most convenient.
Re:hmm (Score:4, Interesting)
I don't believe there is anonymous sftp... (Score:5, Interesting)
What I don't care for with FTP is the continuous setup/teardown of data connections. What is even worse with active FTP is that the client side of the data connection establishes server ports, and the server becomes the client (I'd like to be able to use plug-gw from the TIS FWTK for FTP, but this is not possible for the data connections). However, even when enabling passive FTP, the data connections are too prone to lockup. The difficulty of implementing all of this in C probably contributes to the FTP server vulnerabilities.
Still, if you want both (optionally anonymous) upload ability and access from a web browser, FTP is the only game in town.
From the network perspective, the rsh/rcp mechanism is cleaner (in that there is only one connection), but it still has the problem of either passing cleartext authentication or establishing unreasonable levels of trust with trivial authentication. In addition, with rcp syntax you must know much more about the path to a file, and there is no real "browsing."
Many say that SSH is the panacea for these problems, but sometimes I am not concerned about encryption and I just want to quickly transfer a file. The SSH man pages indicate that encryption can be disabled, but I have never been able to make this work. SCP also has never been implemented in a browser for file transfers. I should also say that I've never used sftp, because it has so little support.
Someday, we will have a good, encrypted file transfer protocol (and reliable implementations of that protocol). Sorry to say, but ftp, rcp, and scp are not it. What will this new protocol support?
Boy, I never thought that I could rant about file transfer software for so long!
USE HTTP!!!!! It is cache-able (Score:1, Interesting)
Re:hmm (Score:3, Interesting)
Re:My opinion (Score:2, Interesting)
Of course, since these days most networks use switches instead of hubs, one has to wonder how much of a security risk sniffers are anyway. I guess there's always the possibility of ARP poisoning or some rogue router between the source and destination, but honestly, how likely is that? I guess it just depends on how secure you need your data to be.
Re:Different, not better or wose (Score:5, Interesting)
FTAM! (Score:3, Interesting)
IBMGSA were running their IT systems, and they decided that FTAM (over OSI) was the way to go, citing FTP's supposed lack of reliability, and lack of features such as resume...
They were absolutely serious when they said that too.....
We eventually managed to get the whole setup working (FTAM, OSI, X.25 over an ISDN link)...but it was the most unreliable, cryptic, complicated system I've ever had to deal with...
A couple of years later, the client ditched IBMGSA, and the company that took over their systems took one look at the FTAM / OSI mess, and threw it out and replaced it with FTP over TCP/IP, and we've never had another problem.
Re:OR, How about... (Score:4, Interesting)
Also, you forgot the first and biggest site with MAGNET links [bitzi.com]. Still, an excellent tutorial, thanks for writing it!
College blocking ftp? (Score:4, Interesting)
Re:Different, not better or wose (Score:5, Interesting)
True, but ftp has far more negatives. The connection model ftp uses, in which the server connects back to the client for data transfers, is a horrible idea. Surely, there must be a benefit to this method (or else why would ftp have become such a standard?), but I've yet to hear it. Passive ftp (PASV) is supposed to compensate for the fact that clients behind NAT can't use ftp, and passive is pretty much supported everywhere now. But passive doesn't solve the problem of many-connections, it just shifts it: With passive ftp, there is still a seperate "data" connection for every file transfer, or transfer of directory contents. It's just that with passive ftp the data connection is initiated by the client to some high-numbered port on the server. This allows clients behind NAT to do ftp, but it makes firewalling the server a pain in the ass, especially if the server is (god forbid) behind nat itself. It's relatively easy to setup a mailserver, or webserver, or ssh server, on a machine behind nat (assuming you can forward ports on the nat gateway). But try doing the same with ftp. Passive connections won't work, unless the ftpd is configured to only use a certain range of high ports for it's data connection, and the firewall forwards that entire range. You'd think with all these connections, the protocol might support multiple connections from one login, ie list another directory while downloading. Well, thats not the case at all, you have to login again if you want simultaneous transactions.
And then if you want tunneled connections... oh boy, what a pain in the ass:
I set this all up for someone recently, and it DOES finally work, with tunneled passive ftp connections to a proftpd server behind a nat gateway, but it was a big pain. Due to the way proftpd works (not sure about other daemons...) the passive connections are directed to the interface the client is connected to. So, with an ssh tunneled connection, proftpd sees the client connection TO 127.0.0.1 (because thats where sshd is). So it tells the client, connect back to me for the data connection, my IP is 127.0.0.1. And the client tries to connect to itself, and fails saying "can't open data connection". The solution was to point the tunnel not to 127.0.0.1, but to the external IP, and have the nat gateway forward internal traffic back to itself. So traffic comes in over ssh, over to the firewallwall (where the nat rewriting happens), and then back to the same machine for ftp.
If people could just use sFTP clients, this wouldn't be an issue _at_all_ (One client connection, one tcp connection! Hey, what a concept!). But due to the integrated ftp clients in programs like bbedit and dreamweaver, I suspect we'll be dealing with that bastard protocol known as ftp for quite a long time....
Damn, reading back over this, I'm kind of ranting. BUT ITS ALL TRUE!
Sorry bout that.
[X] Post Anonymously
Re:No, (Score:4, Interesting)
I have 3.5mbit DSL, but my ISP's performance is flaky. However, I have no problem pulling 300-350KB/s with a download accelerator.
FTP *is* obsolete (Score:2, Interesting)
But the question brings up another point: FTP is a rather cumbersome protocol for transferring files. Ok, so you have two separate connections for entering commands and for sending data. Why then can I not enter any commands while data is being transferred?? Some FTP clients simulate this capability by open multiple control connections, but it really should be built into the protocol, IMHO.
HTTP was designed for small text file transfers, and not really for bigger files (although it does usually handle them pretty well).
I propose an "FTP2" protocol. This protocol will have the following features over and above the original FTP protocol:
Control connection is fully functional even when data is being transferred
Multiple data connections are allowed for each control connection (configurable by server operator)
Ability to queue files for download
Separate server-wide limits on control and data connections (for example, so I can log in and look around even if all data connections are used up. If I try to download something, it is queued until a data connection frees up. This puts an end to hammering on anon-ftp servers.)
Modular authentication -- ability to support authentication methods other than clear-text passwords
Support for encrypted connections
No server connecting to the client to establish a data connection, eliminating most firewall and NAT problems
MIME-type support, rather than clunky BIN/ASCII modes.
What ever happened to fsp? (Score:3, Interesting)
I'm not suggesting this is the answer to your problem, I haven't seen any GUI-based fsp clients (or a unix cli one for ages for that matter).
But it did seem very useful for what it was designed to do, serve out large files to anonymous outsiders.
Re:HTTP is fine (Score:3, Interesting)
My layers are not "messed up" in any way. HTTP and FTP have everything to do with firewalls because the protocols have recommended ports they run on, and HTTP sticks to one port and, thus, can be easily allowed in firewalls. FTP's use of ports depends heavily on the protocol since it specifies the use of both incoming and outgoing connections on various ports for the whole schebang unless you use passive, which is a setting, and by far not the default. Even with passive, the protocol embeds ports and is generally firewall unfriendly to firewalls, hence me saysing FTP is not firewall friendly.
Also, directory listings, as talked about elsewhere in this article, have a lot to do with HTTP as it does not have a specification for them whereas FTP does. It's a protocol thing.
Oh, and you would turn on passive if you had firewall problems, but I'm assuming that to be a typo. Long day for me, too.
Re:how about rsync? (Score:5, Interesting)
Rsync is the way to go!
Rsync is great in theory, but the implementation has one major problem that makes it less than ideal for many cases: It puts a huge burden on the server, because the server has to calculate the MD5 sums on each block of each file it serves up, which is a CPU-intensive task. A machine which could easily handle a few dozen HTTP downloads at a time would choke with only a few rsync downloads.
This is a problem with the implementation, not with the theory, because it wouldn't be that difficult for the rsync server to cache the MD5 sums so that it only had to calculate them once for each file (assuming it's downloading static content -- for dynamic content rsync will probably never make sense, particularly since we can probably expect bandwidth to increase faster than processing power). The server could even take advantage of 'idle' times to precalculate sums. Once it had all of the sums cached, serving files via rsync wouldn't be that much more costly in terms of CPU power than HTTP or FTP, and it would often be *much* more efficient in terms of bandwidth.
Re:Different, not better or wose (Score:5, Interesting)
Re:hmm (Score:2, Interesting)
I have not had a problem with IE on ftp servers, but then again, I only use it at work. I would think that both would be the best choice though.
Re:hmm (Score:3, Interesting)
I have never seen any evidence that this is true. Why should it be true anyway? After the http headers have been sent, you're basically just sending the file's data straight through a plain TCP connection. What could be simpler?
As for IE's ftp problems, maybe they only manifest themselves if you have "folder view for ftp sites" turned on in the preferences. I do, because I sometimes want to upload/move/rename files as well as download and I like the drag-and-drop folder interface. But IE often freezes for almost a minute while it tries to contact ftp sites. I'll have to try turning that option off. I just wish Phoenix had a method of uploading/renaming files on ftp sites. Maybe some genius can point one out to me.
Re:hmm (Score:4, Interesting)
While you have some valid points, IE handling ftp poorly is not really a problem with FTP. A wrench makes a bad hammer.. that doenst make nails any less useful.
I havent had to worry about continuing a download since i stopped using my 2400 baud modem. The largest problem with downloaded are apps that autodownload (updates) and dont handle resuming. On broadband I havent concerned myself with an interrrupted DL in several years.
As for the firewall issue, if your admin is allowing downloads via http and not ftp for alleged 'security' reason he/she/it is a retard.
To address security holes, if there have been anyproblems with the ftpd's lately they dont get a lot of press. If you are referring to IIS, well... thats your own lookout.
Re:WebDAV + HTTP (+ SSL) (Score:3, Interesting)
What I wanted to do was this; I'm working at the office. I have my server running at home. I want to take some work home with me, so I pull up the WebDAV server and drag-and-drop those files from my desktop. Voila! Once I get home, there they are.
Problem is, I can't do anything with them. I'm not supposed to modify them directly, nor am I supposed to copy new versions to the WebDAV directory (to download once I get back to work).
The only way I can do it (according to the mod_dav people) is to make copies of the files into a work directory, edit them, and then use my browser to copy them again via WebDAV, to localhost. Seems goofy to me.
If this has changed with some more recent iteration of mod_dav than the one I used, let me know. But this isn't the only reason that WebDAV strikes me as still not-quite-ready for prime time.
1-6MB files to how many users? (Score:3, Interesting)
Hmm.. There's no reason why you couldn't just perform the download via HTTP if your target customer or audience has the bandwidth.
If you're concerned about the customer losing connectivity, you could always just break the file up into small chunks for download.
Both FTP and HTTP, depending on clients and servers, supports continuation-downloading(ie, recovering from where you left off.) So if one connection flakes out, you can always recover.
The long and short of it?
HTTP(S) or FTP(s/FTP), it doesn't matter. With both, you can restrict access and/or grant access to specific files. You can recover from some point or offer split files. With HTTP, you can have the information piped through a back-end piece of code so that you can monitor who does what when, but something similar can be done with FTP as well.
If you've already got a webserver going and secured, you're probably better off offering the files via HTTP. Nothing unprofessional about it.
Fact is, if your target audience is the general public, you'll fare better with HTTP. Why? Because while most people who are online will probably have access to a web browser and know how to download HTTP files, they may not have an FTP client or necessarily know how to use their browser as an FTP client.
Me? I offer photos and docs online via HTTP. It's easier and I can control the visual aspect of the "download area" to create a more professional look than that of the "industrial" FTP interface.
Nothing wrong with FTP or sFTP. I use them at work and at home. But then again, when I find a site which offers HTTP or FTP? I pick HTTP to download first. It's only when I'm still looking for something that I go with FTP... which, by the way, becomes difficult to navigate on some clients. A nicely generated webpage featuring the item via HTTP would be much more elegant.
Just my two cents.
EVIL FTP PROTOCOL FROM HELL MUST DIE! (Score:1, Interesting)
Note I'm talking *protocol*, not any particular client or server implementation. The protocol is fundamentally flawed.
I'm dealing with this decision now too (Score:4, Interesting)
So far, it's been a failure for two reasons:
1. IE blows as an FTP client, and users aren't comfortable dropping into the (somewhat crappy) DOS FTP client.
2. Firewall setups at the fortune 500 companies that we deal with normally seem to keep FTP access off-site restricted.
Re:Different, not better or wose (Score:4, Interesting)
Ever heard of web page directories that almost every common web server uses, and most web servers automatically do when no index file is present?
I find lynx+ web page directories a lot faster to get files across my network than ftp.
The only real advantage I see with ftp is uploading files quickly and easily... but that's not for me
This is Bad and Rong (Score:2, Interesting)
In Unix, you can background this operation fairly simply...
This will start downloading your file in the background, will autoresume if something happens, waiting 15 seconds between retries, and never give up until the file is completed or the source no longer has it. You'll get an email when it completes one way or another.
Re:Forget them both.... Anonymity (Score:2, Interesting)
Also, if you think that just knowing when data packets are sent and received tells you the contents of the data, then why hasn't SSH been broken yet? With those assumptions, it should be easy to use a statistical model of people's keystrokes and break into a session. Realistically? Not possible.
You've got to start somewhere, especially if you're going to get the world to transfer to using encryption everywhere. The mail server I use for private mail among friends has no unsecure channel going out of it. All of it's protocols are protected with SSL/TLS, including SMTP inbound and outbound. And besides, the argument I was making was for using encryption to connect to a public mirror site. Since the majority of mirrors have HUGE numbers of different files, and the amount of control data sent is never constant, knowing how much data is sent in either direction will NOT be a guarantee that you know what was transferred.
AND, I was talking about legal prosecution (or persecution, depending on how you see it), and a document that says you downloaded approximately enough bytes to be a suspect program isn't enough to stand up in court...
Re:how about rsync? (Score:1, Interesting)
But if the recipient doesn't have the file already rsync wouldn't need to do incremental checksums anyway.
(I love rsync, I mirror changes on a 250 gig file server every night, runs in 20-30 minutes generally (5-10 gigs of new stuff))
What about rsync or scp (Score:2, Interesting)
In the uploads arena, I could transfer any file faster on rsync or scp, or even cvs -z5. And thats over 56k.
FTP isn't so secure, it's pratically text on port 23 (wonder why the warez sites still use it, aren't they trying to avoid goverment agencies? and M$ and all that crap?).
HTTP is faster for smaller files as FTP eates up a whole lot of time doing the connection procedure.
Re:"Files," eh? (Score:2, Interesting)
Re:HOW ABOUT UPLOADING??? (Score:1, Interesting)
FTP a crumbling legacy protocol? (Score:2, Interesting)
They are two different protocols designed for two different purposes. If you plan on transfering large amounts of data, and want it to be more reliable, I would most definetely recomment FTP. FTP is made for transferring files, and that is what it is best at doing.
If you are not concearned with data integrity and plan on transferring decent sized files, I would wish you the best of luck with HTTP.
As far as security holes go? I have no idea, but I imagine both protocols have had their fair share of security issues.
For more reliable transfer of larger files, I would really recommend FTP.
Re:hmm (Score:2, Interesting)
Web browsers don't support reget, AFAIK. (Please correct me, if I'm wrong.)
If John Doe is a little bit educated, he might use tools like GetRight. GetRight supports partial transfer from both FTP and HTTP.
MGET (Score:2, Interesting)
ftp includes mget. So I can download all the pictures with:
mget *.jpg
Without messing around clicking and saving lots of things. Sometimes a command line rocks.
Andrew
P2P! (Score:3, Interesting)
(yes, yes, I know, requiring people to use spyware-laden software is not nice, but it's a good idea in an ideal world)
Limit number of connexions, NOT which FTP client (Score:4, Interesting)
I can make a dozen or more connexions to your FTP server with nothing more exotic than Netscape. Why pick on download managers when they use the same number of connexions? (BTW, Getright says right in its configuration that "some servers regard segmenting as rude" and recommends against it.) Better to limit connexions to x-many per IP address, and let the user spend them any way they wish.
BTW, if you do limit connexions, please remember that it usually takes one for browsing the site (using Netscape or whatever) PLUS one for the download manager to fetch the file. Otherwise the user who was looking with a browser has to leave the server, then wait for the browser connexion to close (which can take a while) then finally paste the link into the DLManager. So a limit of two connexions from a given IP is a nice practical minimum, and surely not a hard load for anything outside of home servers operating over dialup.
PS. I love FTP's convenience, and I always try to be extra-polite to small servers (and not rude to big ones). I do use Getright, and have segmenting disabled (which BTW is the default).
Re:HTTP is better (Score:3, Interesting)
Neither has HTTP.
"Bad error recovery, many differing standards on restart"
Erm, yeah, it's so non-standard that prozilla doesn't exist - exploiting the `resume' potential of most FTP servers to parallelize downloads starting 0, 25, 50, 75% through the file. Yeah right.
"Designed for human readable command lines not machine readability"
Oh, come *on*. When there's no standard output format at all in HTML, how can an `ls -l' output not be vaguely standard? Are you unaware of the SIZE FTP command?
"Password sent in plain text, not even concealed"
And somehow HTTP with its sending of usernames and passwords also in plaintext is "better"?? Does it look flourescent pink on your screen, or something?
"HTTP: none of the above"
Poster: no clue what he's on about. HTTP: all of the above. Readers: much annoyance.
HTTP is more easily integrated with SSL in the form of HTTPS.
Almost everyone has an ftp client. You have to install curl or wget or lftp yourself consciously in most distributions.
HTTP: no standard for file uploads, you rely on server-side configs and handlers for the purpose. How mad is that??
Re:Limit number of connexions, NOT which FTP clien (Score:2, Interesting)
that'd mean only one user in the entire company can use the server - bad.