Laptop/Server Data Synchronization? 305
gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync.
The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"
Windows might be good for something (Score:4, Interesting)
My Briefcase in Windows 95. It even has a cute ickle briefcase icon.
Somewhat seriously, Offline Files in Win2K/XP is something I've yet to see done well on any other OS.
Re:Subversion (Score:5, Interesting)
At the risk of saying something stupid or blasphemous: why offer something that requires "writing some scripts"?
If the OP wanted to "write some scripts" s/h/it could have done all the neccessary work with a couple foreach...cp...end. Or, hey, rsync.
I am suspecting the OP is wondering whether there isn't something out there that "just kinda works" and only needs intervvention in case of a conflict.
Knowing well that this will definitely be considered blasphemy: I've been using Window's "briefcase" system since Win98. It does "kinda work". Most of the time. And requires work when there's a conflict. Which appears to be what the OP is looking for. Given that the OP doesn't seem to want to just go that route, the question appears pertinent what s/h/it is looking for that Mr. Gates briefcases can't/won't do...
Groove (Score:1, Interesting)
Try coda (Score:2, Interesting)
commercial products. (Score:4, Interesting)
Novell's iFolder is a very interesting alternative... runs on Linux/Apache/Java stack & only transmits changed blocks over an SSL connection.
Other things worth looking into include Microsoft Groove--let's you synchronize an entire workspace with yourself on other computers or other people - and is relatively network & environment-independent (though Windows only)
Re:Windows & Make Available Offline (Score:1, Interesting)
When I tried using it I found it made the system intolerably slow when resuming from standby or placing the laptop in a dock or anything that made the system try to resync the files.
And then it was just plain wrong about determining which files needed syncing.
At the time I did this, about a year after XP came out, it was clear from usenet groups that the windows solution was woefully tested and just not ready for prime time.
It was then that I started using Unison, which when combined with ssh tunneling makes for a great solution.
Re:iFolder? (Score:5, Interesting)
I don't know how mature the novell version is but the open source version is very far from being mature. In fact there hasn't been a stable release in more then two years and nobody knows if or when there will ever be a stable release.
A while ago all the developers on ifolder either quit or were fired and the development was moved to india. Since then the pace of development has slowed down to a crawl and the new developers try to understand the code base and fix bug reports.
Right now you can download something that is beta-ish but I certainly would not trust my mission critical data to it. If you want something that works you are going to have to pull off the trunk and compile it.
Synchronization Woes (Score:5, Interesting)
The more complex problem has been thrown at me a few times. What if it's not just one person?
Say you have a repository of data that a dozen people may be working in. When they're all network connected, they're all dealing with the same file pool. When they take their off-line copies with them (unplugged laptops on vacation), they all make changes to the same files. Maybe mine is a one line change. Maybe one guy copy&pasted the first 3 chapters from War And Peace into a comment somewhere in the middle. Maybe another developer did some very intellectual looking changes but hosed some major functionality.
When you start putting machines back on the network, who is right? The 6 guys who did real work are obviously right(ish), but they all made different changes. The very last change will end up being someone's 3 year old kid who was pounding on the keyboard right before daddy shut down the laptop, saving the new changes. Probably the last is the most recent, and right by most methods.
It's not a pretty picture, and requires some intelligence to sort out the mess.
The only "good" resolution I've found is to give logical authority to the changes. Bob is in charge of development. Any changes going into the development or production tree must clear him. He should be able to recognize that the 6 guys made changes, and diff them to come up with the common changes. The 3 chapters of war and peace go by the way side. And the guy with the 3 year old "developer" gets reprimanded.
In the end, a good revision system and good backups are needed too. Something will slip through the cracks, and you'll need to roll back to something you hope is good.
I take control over whatever I'm working on, so if I know I'll be working offline, I'll scp the data to my laptop, work on it on the road, and scp my changes up to the server when I'm done. Anyone else who may have worked in my project space in the duration should have known better.
Re:Coda (Score:3, Interesting)
I've always wanted Coda to work for this, but I haven't ever gotten it working. My current thinking is that I'd need to set up a server on my laptop, and then have the client talk to the local server. Then the two servers could sync.
Does anyone have any information or case studies on how to make this work for a small network? Easy conversion tools? It seems like the ideal solution to me, but getting it to work seems difficult.
--Ted
Re:Windows & Make Available Offline (Score:2, Interesting)
I use it with my wife for our financial data. She syncs, makes changes, then next time I connect I can choose to use the server version or my version. Since there is no version control, you have to communicate. Then again, with the way version control works, if you end up merging a lot, perhaps too many people are working on the same problem anyhow.
In the end it works well.
Re:Coda (Score:2, Interesting)
I haven't actually used Coda, though I'm planning on it for a small network myself, because this is exactly what it was designed to do. But why would you need the server on the laptop? All you should need is a client. Have you tried it that way and it didn't work or something?
I do hear it's a pain to set up though.
Re:rsync (Score:3, Interesting)
As far as the user is concerned his primary concern for laptop data loss is email. So you have to back it up as a part of any backup solution. If you are storing email locally on the laptop and backing it up the backup will nearly always be corrupt due to file locking. Further to this, the financial, network and storage resource required to implement a laptop backup solution exceeds by far the cost of extra storage on the mail server. In addition to that if the email is stored centrally the backup size per laptop decreases straight away to become on the order of MBs instead of GBs for most users.
Re:rsync (Score:1, Interesting)
Re:rsync (Score:5, Interesting)
- syncs continuously/watches the FS for file changes (no cron jobs needed -- things usually sync as quickly as they can be sent)
- does binary diffing and only sends deltas (compressed & over SSL)
- transparently archives past versions of all files (i.e. undelete/infinite undo)
- syncs across any number of machines
- lets you get to your files from the web
- some more info @ http://venturebeat.com/2007/08/16/the-y-combinato
We made it after hacking together our own rsync-based abominations and getting really annoyed that no one had solved this genre of problems in a way that normal people could use.
Okay, I can stop shilling now. I was just excited that other people run into these problems.
Re:rsync (Score:3, Interesting)
This looks quite similar to Novell's iFolder but with you running the server yourselves instead of having your users set up an iFolder server. Last I used iFolder was in the 1.x or 2.x days and it frankly wasn't anywhere near the polished product you have here. Now it seems that iFolder 3.x [ifolder.com] is open source and looks a lot more polished.
Still, I think you surely have a great service market here even though the polished front-end app seems to now be done open source. Best of luck to you on your new venture!
Re:Unison, Rsync & NTP (Score:3, Interesting)
Will there be a Linux client? (Score:2, Interesting)