Distributed Filesystem for Disconnected Operation? 58
juraj asks: "I'm trying to achieve the following setup: I have two offices connected via a relatively slow ADSL line, and I want a shared fileserver between the offices. I have VPN using IPSec ready, so security is less of a concern, but simply mounting a filesystem (via Samba or NFS) from one office to another is not a solution because of the speed. Also, the ADSL line is sometimes not only slow, but also disconnected.
I've tried the CODA distributed filesystem to achieve replication, so that both offices have local copies of their files. The problem is, that the CODA filesystem is just a research project: it is unstable, with the venus daemon constantly falling, and sometimes when recovering from the disconnected state, one side does not recognize the changes and they are simply not propagated.
Have you had any good experiences with CODA? Which versions do you use? What kind of setup did you have? How is it configured? I've also heard about OpenAFS, but similar to CODA, I've learned it is unusable in a real environment. Is there any real solution to my problem? Are there any decent solid free distributed file systems for Linux or the BSDs?"
This question (Score:4, Interesting)
Both servers running at the same MX. So users could choose the server 1 or server 2 according to the location. And witch among them, in case one network goes down.
Although it sounds simple, I don't know any simple solution to that. Rsync won't work, as there wouldn't not be a master server. Both would have the same preference, so no server depends on the other. That's the goal.
Re:This question (Score:5, Informative)
I developed a replicated filesystem that we use with our commercial email service. The filesystem is layered under UML (User Mode Linux) and cross-replicates files between two servers, on in California, and one in Pennsylvania.
I too looked at Coda and Inter-mezzo, but was not very satisfied with their stability and/or their ability to recover from outages.
The replication that we use relies on the update nature of MailDir with Courier Imap.
Our solution uses UML to post a transaction journel to the underlying host OS layer. Application level code then cross-posts filesystem updates using HTTP transactions with curl and Apache/cgi. Transactions are delayed about 2 seconds to coalesce multiple updates into a single network event. In general, we get about 5mbit of update thruput coast to coast and it is very rare that either system is more than a couple of seconds out of sync.
I am sorry that I cannot give you the code. While the code is Linux bases, we don't actually sell (distribute) it, so we keep it in-house for our own use. Perhaps my description will give you some ideas.
The email offering is described at:
http://easyco.com/mail/index.htm
Re:This question (Score:2)
Also, what am I missing that makes it significantly better than a triggered rsync session, i.e. one that runs either periodically or whenever a threshold of changes is exceeded?
Re:This question (Score:2)
The actual file-system driver in UML is patched to produce an "event log" for all file updates. Because this is UML, this log can run in user-space to the host OS.
The log itself includes events such as "close file after writing", rename file, create directory, set priviledges, etc.
The log itself is sent to a pipe. The pipe is dequeued by a daemon that builds transaction "blocks". A transaction block can contain one or more transactions. When a transac
You can use rsync (Score:1)
Re:You can use rsync (Score:2)
Re:This question (Score:3, Informative)
MOD PARENT UP (Score:1)
Re:This question (Score:2)
intermezzo! (Score:4, Informative)
you are looking for intermezzo
http://www.inter-mezzo.org/
the same guy from coda is the leader. remember
afs -> coda -> intermezzo
Re:intermezzo! (Score:4, Informative)
My company uses redundant leased lines to home (different breeds and providers) to ensure that every building can access network resources at all times. Manual fail over. We're not a huge company, but we manage most of this in-house. We'd *love* to know if there's a better answer, even if it cost a lot of money.
Well. There's always a better answer on the other side of a long and expensive implementation process.
Re:intermezzo! (Score:2, Interesting)
Re:intermezzo! (Score:2)
Re:intermezzo! (Score:1)
I could be wrong, but I think if you've got two versions of a non-text document being concurrently modified in separate locations with the net down, you're going to have issues no matter what file system you're using.
Re:intermezzo! (Score:2)
Re:intermezzo! (Score:3, Informative)
you are looking for intermezzo
Hmm.. let's just look at the mailing list again... maybe just a snip from a recent(ish) post (Mar 22 - there are 8 posts since this one, half of them spam):
| Don't post to a list without reading it also.
I read this list, what little of it I get.
| And don't complain about the state of open source
| software, if you are not ready to test it's betas.
I am certainly ready to test the betas, just that the last time I tested
Intermezzo or Lustre -- Lust
Re:intermezzo! (Score:1)
Re:intermezzo! (Score:1, Offtopic)
Unison (Score:5, Informative)
--
Evan
Re:Unison (Score:2, Informative)
Novell ifolder (Score:5, Interesting)
It's opensourced even and available on Novel Forge.
Re:Novell ifolder (Score:3, Informative)
Perforce (or any other Version Control system) (Score:5, Informative)
From my experience, Perforce [perforce.com] has the best use of bandwidth and also the most intelligence when it comes to rearranging directory structures and resolving conflicts.
Unfortunately it's only free [perforce.com] for up to two users - so it may be useless for your needs.
Re:Perforce (or any other Version Control system) (Score:5, Informative)
I have used P4 (perforce) to keep a lot of files in sync between two locations. Fortunately, I had only two locations, so the 2-user 2-client limit never was exceeded.
In case you want more clients/users, you can try for any of the following:
1. CVS (http://www.cvshome.org/)
2. GNU Arch (http://www.gnu.org/software/gnu-arch/)
3. SubVersion (http://subversion.tigris.org/)
All these are excellent source control tools, and operate over ordinary TCP/IP (don't need a special setup).
Avoid tools like Visual SourceSafe because they require a network-mapped drive to work.
http://better-scm.berlios.de/comparison/compari
Re:Perforce (or any other Version Control system) (Score:3, Informative)
OpenAFS unusable in a "real" environment? (Score:5, Informative)
I'm sure this thing has more than surpassed terabyte size by now. It was always fast and always reliable, except when the one of server's SCSI cards would melt and start spewing errors.
AFS is better than most people give it credit for. I'll admit, it isn't easy to set up, but all the features that you get for that initial work are well worth it.
Re:OpenAFS unusable in a "real" environment? (Score:5, Interesting)
Re:OpenAFS unusable in a "real" environment? (Score:3, Informative)
As far as Kerberos goes, I'd suggest the new ORA nutshell book "Kerberos: The Definitive Guide". While it doesn't go into AFS much, it explains how the thing really works and how to configure MIT and Heimdal Krb5.
- Happy AFS/krb5 site administrator
Re:OpenAFS unusable in a "real" environment? (Score:1)
Re:OpenAFS unusable in a "real" environment? (Score:2, Informative)
I'll be happier once the stable versions have two things though... >2gb file support, and support for 2.6 series kernels. Disconnected operation would be nice as well.
All of those are proposed projects, but not currently in the developement version (at le
AFS doesn't work at my uni (Score:1, Informative)
Monash University [monash.edu.au] is using AFS on its Linux desktops [monash.edu.au]. Whenever the connection to the file server goes down, everyone's sessions hang, which is clearly unacceptable.
It's quite possible that it has been incorrectly set up, but in this situation AFS hasn't delivered what it promised.
Re:AFS doesn't work at my uni (Score:2, Informative)
Re:AFS doesn't work at my uni (Score:1)
Re:AFS doesn't work at my uni (Score:2)
AFS works fine (Score:3, Insightful)
Among the most notable of the universities using afs are CMU, UNC, RPI, MIT. Furthermore, there are a number of government namespaces as well.
Try it out!
Re:AFS works fine (Score:1)
Lots in other countries too:
% ls /afs
afs.hursley.ibm.com eos.ncsu.edu mcc.ac.gb sfc.keio.ac.jp afs1.scri.fsu.edu es.net md.chalmers.se si.umich.edu alw.nih.gov ethz.ch me.cmu.edu sipb.mit.edu andrew.cmu.edu federation.atd.net meteo.uni-koeln.de slac.stanford.edu anl.gov fh-heilbronn.de mpa-garching.mpg.de sleeper.nsa.hp.com asu.edu fl.mcs.anl.gov msc.cornell.edu spc.uchicago.edu athena.mit.edu fnal.gov msrc.pnl.gov sph.umich.edu bnl.gov geo.uni-koeln.de msu.edu spv.uniroma1.it bp.ncsu.edu glue.umd.edu na
A similar question: edit/compile/run over ADSL (Score:1)
Re:A similar question: edit/compile/run over ADSL (Score:3, Informative)
Edit at your local site, have a (subversion/cvs)server at the office.
Re:A similar question: edit/compile/run over ADSL (Score:2, Informative)
I mount via nfs over a VPN over 1Mb ADSL (rsize=8192,wsize=8192,intr,rw,async,noatime,noau t o,user) and after the Vim session is restored, don't have a problem.
An rsync based script (FWIW in Python) to xfer disparate directories and files works around the cumbersomeness problem.
As for the 'use version control' responses: I don't want to store intermediate versions of persistent files and don't want to store intermediate/temporary files at all (but don't want to recreate them from scratch every couple
Re:tramp (Score:3, Interesting)
Unison (Score:5, Informative)
Re:Unison (Score:4, Interesting)
cvs (Score:1)
Just a first thought...
Re:cvs (Score:2, Informative)
You should use CVSup [cvsup.org] for this.
It has already proven its useability for syncing and updating FreeBSD systems
I think you _could_ use rsync (Score:2, Interesting)
http://lists.samba.org/archive/rsync/2001-
use AFS (Score:4, Interesting)
Re:use AFS (Score:2)
I'd love to get started with a supported AFS... is this something that only corps can buy?
Jonathan
Re:use AFS=OpenAFS (Score:2, Interesting)
This means I will probably check it out for my next fileserver project...
Re:use AFS (Score:4, Informative)
Oracle iFS? (Score:2)
Foldershare (Score:2, Informative)
I read [deviantart.com] that "the development team hopes to start work on Mac OS X and Linux clients within the next six months" (Jan 27th 2004).