Laptop/Server Data Synchronization?

gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync. The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"

my take

[TheRealMindChild]

Man... You are late to the party. People have been struggling with this since the beginning of time (or so it seems). Especially database apps, where they need to work in "detached mode".

I can't give you a flat out solution, because all situations are different. But I can pass on a bit of wisdom. The most important thing for you to do is create business rules for your synchronization. If the data on the server has changed and you made changes offline, who gets priority? You will have three categories of which a file can be... Client changes get priority, Server changes get priority, and Merge files. I would stay away from the last one. If you want to keep things simple, Id go for the "Server changes get priority" approach. In short, if you took an "online" file "offline" and came back, and the server copy has changed since, your offline edits are abandoned. This way, it makes it so heavily edited files have a shorter "check out window" (even if you don't use a checkout system), and forces the person taking the file offline to coordinate with everyone else that may edit this file.

common problem

[ILuvRamen]

When the laptop is reconnected, I want the data to automatically re-sync. The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop.

And there you have the problem with synchronization. There's no mind reader program (yet) so sorry but you're going to have to make up your mind about how to handle it when the server version changed too. Either find a way to merge the files or start making decisions about when they can get modified (ie a checkout system) or if the server's or the laptop's version is always right or if the user gets to choose (bad idea). As for programs, stop looking at famous ones, they suck. That's like using Norton as your antivirus. Find some freeware or open source one that does just what you need and isn't overly complicated

Am I missing something?

[aero2600-5]

From the summary:

"The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop."

So let me get this straight.. You have the old version of the file, somewhere. The new laptop version of file, somewhere. And the new server version of the file, somewhere. And you want the software to decide which to use and copy it to both the server and the laptop?

There are even more issues here, but it kinda sounds like you want some artificial intelligence that you can download.

Aero

Coda

[norkakn]

http://www.coda.cs.cmu.edu/ [cmu.edu]

OS X Client & Server does a good job with this

[phillymjs]

I'm not exactly sure what Apple uses under the hood to accomplish it. I don't think it's rsync, because I've fooled with the rsync built into OS X and I get errors frequently, but their home syncing works great.

When you have a mobile user account (i.e. a network account with a local copy of the home folder on the workstation), it will sync every so often (frequency and exactly what is synced/skipped can be configured on the server end, and the user can kick it off manually from the client end). To the best of my knowledge, the sync is bidirectional, so if you log into another machine with a mobile account and modify the server copy, the changes will be reflected on the mobile copy at next sync. It makes my life easier because if a laptop user's machine gets lost, stolen, damaged, or destroyed, we've automatically got a backup copy of the data on it up to the last time it was synced.

In the event of conflicts, the user is presented with a dialog asking which version to keep, including file size and modification date.

Note that I'm not suggesting you throw out your existing hardware and buy Macs to get this feature, but maybe look into exactly how it's done on the Mac and see if you can duplicate it on your systems.

~Philly

Re:rsync

[cduffy]

subversion is intended for a case where you have a single data store. A modern distributed SCM -- designed for disconnected use -- would make more sense.

Personally, I play with bzr most frequently; it has a nifty Python interface (and an extensive plugin architecture) which makes it quite conducive to local scriptage. (As an example -- I have a local, filesystem-backed set of CA scripts which use bzr for transactional semantics; if a method is called which throws an exception, all filesystem changes are automatically rolled back; if a method succeeds, a commit is done to record the operation and [effectively] set a rollback point). The separation between working tree and repository is optional (by default, all working trees are also repositories -- much like BitKeeper in that respect), which makes it very handy for situations like this where you don't necessarily *have* a separate, central location which all nodes can always communicate with, and where the different trees are allowed and expected to temporarily diverge.

Re:Unison, Rsync & NTP

[Colin Smith]

Sure there is another way: newest file wins.

But this means that any number of intermediate edits by arbitrary numbers of people will simply and silently be removed if someone updates an obsolete version between syncs.

nevertheless I couldn't find software that would support this, thus had to write it by myself...

Eh? Rsync supports it. rsync -u ...

 

Re:Subversion is for stupid and ugly people

[cduffy]

git's chunk-based rename handling is interesting, but bzr's directory-level handling is closer to what most users expect. (Does the git behavior make more sense for the kernel source tree? Sure! Does it make more sense for Joe Blow's home directory? I'd need some convincing there).

Out here in the Real World, folks setting up revision control systems need to count "stupid people" (read: artists and web designers who are too busy making art or designing web pages to care about revision control except inasmuch as it's a way to back up and distribute their work) among their customers. For a great many cases, subversion is Good Enough, and it has excellent Windows support, integration with just about every IDE under the sun, TortoiseSVN and other nice pretty hand-holdy tools available which simply aren't ubiquitous among SCMs written with hardcore users in mind (seemingly to the exclusion of those that aren't). SVN isn't distributed, which sucks. SVN has an ugly hack of an excuse for a rename handling algorithm, which sucks. SVN is slow as molasses compared to some of its competition and lacks merge tracking (and thus history-sensitive merging) and has a gawdawful working tree library and sucks in any number of other ways -- but it is a compelling replacement to CVS, and sometimes that's what the customer needs, no matter what shiny happy features ${YOUR_FAVORITE_SCM} may have and no matter how many ways SVN manages to annoy the power user.

And trust me, I learned this one the hard way.

As for the 4NT copy suggestion, the whole bidirectionality and rename handling arguments come into play.

Re:Windows might be good for something

[juanhf]

windows' built in offline files is great; i only started using it in the last year or two and i feel like i have come from hiding and into the light.

the system is configurable through group policy which makes it easy to setup the system throughout an enterprise.

here are some links:

how to use offline folders http://support.microsoft.com/kb/307853 [microsoft.com]

in your case, there is a way to fix offline folder problems (when and if they occur) have a look at:

how to re-initialize offlines files cache: http://support.microsoft.com/kb/230738/en-us [microsoft.com]

last but not least, here is a link to group policy for offline folders: http://technet2.microsoft.com/windowsserver/en/lib rary/9dba5df2-0359-4fa4-bdcf-dd6ae5ca345e1033.mspx ?mfr=true [microsoft.com]

Re:Subversion is for stupid and ugly people

[rawler]

There's a reason why SVN don't distribute well. It simply don't branch well.

And in branching, most of the GUI-users just don't get a clue and practically eliminates any chance there was for decent branching/merging.

SVN/Tortoise are good for one thing, snapshotting, and that should better be handled by the filesystem itself anyways. (Why are filesystem snapshotting STILL not mainstream, btw?)