Laptop/Server Data Synchronization?

gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync. The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"

rsync

[jshriverWVU]

I do this often and rsync is wonderful for such a task.

Subversion

[hahafaha]

How about Subversion? You'll have to write some small scripts that would make it autoconnect, etc., but it could work if you set up SSH keys, etc.

iFolder?

[belly69]

That sounds exactly like what Novell's iFolder is made for:

http://www.ifolder.com/index.php/Home [ifolder.com]

Unison

[graphicsguy]

I use unison. Why would you need to run it manually every time? It can be run in batch mode. I am mostly using it for live backups these days rather than true bidirectional synchronization, so I could really use rsync and some scripts, but I've gotten pretty comfortable with unison.

Re:rsync

[pixel.jonah]

I'd second rsync.

I'd also take a look at Microsoft's SyncToy [microsoft.com] if you're on win***s.

Re:iFolder?

[Anonymous Coward]

Agreed. I work for a little company with a few hundred on-the-road consultants who keep their lives (gigabytes of data) on their laptops and on desktops at work. The workstations are on when they are there and sometimes even when they're gone, and the laptops are on sporadically. When logged in everything synchronizes once per minute. If changes happen on the laptop they go to the server and, if the workstation, they also go the workstation. The synchronization is bidirectional and happens with the newest versions of the files no matter how many clients. We used to use regular "shares" in windows (AD) but dropped it when we added this technology. It's still available for some purposes but our IT department doesn't even use them anymore when iFolder is available because it's faster, automatic, and seamless.

Windows & Make Available Offline

[goofy183]

I'll likely get buried but here it goes:

In Windows you can mark a folder on a network share as "Available Offline". Windows will copy all of the files to the local HD and if the server isn't available just work with the local copies. When the server is detected Windows will automatically sync the files and pop-up asking the user about conflicts (keep local / keep remote). When connected writes automatically go to both the local copy and the server.

One of the few places that Windows has right and I haven't found a Linux or OS X solution for that is nearly as nice.

Foldershare

[Offtopic]

I've been using Foldershare for several months now to synchronize several folders on three different machines. It has worked well so far and it is free. It's available at: https://www.foldershare.com/ [foldershare.com]

Re:rsync

[rebullandvodka]

I agree, subversion & scripts sounds like a good solution. Dare I suggest... Lotus Notes/Domino? Seriously, the notes databases can be setup to do what you're looking for. It worked pretty good at my last job as long as everyone played nice.

Re:Am I missing something?

[sholden]

Have fun trying to merge changes in an executable binary file.

Not seeing the difference between arbitrary files on a disk and files that have been explicitly version controlled is I guess what makes you the hardware guy - does that mean you nail the floorboards down?

Re:rsync

[eos_buddy]

If the laptop and the server you use is Linux/Unix, rsync is definitely the answer to it - its robust and after the 1st time of rsyncing, the process should be quick enough (assuming you log-on often). I wrote a small script recently to sync my firefox bookmarks. Don't know whether it might be helpful to you, but here is the link: http://oidw.blogspot.com/2007/08/geek-talk.html [blogspot.com]

Re:rsync

[Roarkk]

rsync is part of the answer. If you're looking for a way to have multiple, incremental backups of laptops with unpredicatable patterns of connecting to the network, BackupPC [sourceforge.net] is the way to go.

BackupPC is a high-performance, enterprise-grade system for backing up Linux and WinXX PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain. Given the ever decreasing cost of disks and raid systems, it is now practical and cost effective to backup a large number of machines onto a server's local disk or network storage. This is what BackupPC does. For some sites, this might be the complete backup solution. For other sites, additional permanent archives could be created by periodically backing up the server to tape. A variety of Open Source systems are available for doing backup to tape. BackupPC is written in Perl and extracts backup data via SMB using Samba, tar over ssh/rsh/nfs, or rsync. It is robust, reliable, well documented and freely available as Open Source on SourceForge.

By using pooling and compression, one client of mine is using BackupPC to backup over 1TB of data distributed among over 100 laptops to a 200GB filesystem on a central server. The network is polled every hour, and any system that hasn't been backed up in the last 24 hours is queued. Beautiful system.

In OSX, portable home directories

[shamborfosi]

I have OSX laptops using portable home directories to do exactly what you are asking for.. a network home directory that is automagically sync'd to my laptop (thus making it portable). It works both ways, and I'm definitely happy with it. I'm not sure which OS you're using though. I wrote about how to do it in an article: Full Stack: Portable Home Directory over NFS on OSX authenticated via OpenLDAP on Debian Linux [mattfleming.com] if you're interested. I also just got everything to work over AFP to an OSX server running open directory as well.. but haven't had time to write it up yet (btw, a lot fewer steps).

Re:iFolder?

[freckledp]

I used to work for Novell, traveling two or three times a month. My personal office had several computers and I had multiple laptops that traveled with me. Using either Windows or Linux clients of iFolder, I was able to easily sync files from one machine to another, always ensuring that I had the latest copy of whatever file I needed. It sounds like exactly what you need.

One possible problem: you have to store the information in a folder (which you specify). Only the data in that folder is synced.

HTH

-FreckledP

Re:Windows & Make Available Offline

[Techman83]

A great solution till it breaks... believe me it does break and when it does be prepared for heartache. There are ways to recover it, but I think it assumes to much and the potential to screw up is a big risk. There were several users at my place of employment that found out this the hard way and now we ban the usage of it. It's not so much finding the best tool, but managing the process overall and how to do that.. Well we are still in the process of developing that one!

Unison

[sgyver]

Maybe a two-way rsync tool made just for this purpose?
You might have to do A-B, A-C, A-B type syncs for more than 2 paths, unless you stick to a hub/spoke or cascading distribution model.
Not all conflicts are automatically resolved, by default.

http://www.cis.upenn.edu/~bcpierce/unison/ [upenn.edu]

Good luck.

Re:rsync

[frenetic3]

Take a look at Dropbox (http://getdropbox.com/ [getdropbox.com]; screencast at http://getdropbox.com/u/2/screencast.html [getdropbox.com]) if you want something that's rsync-like but integrated into Windows and OS X. It's in beta (and full disclosure: I co-founded the company) but was designed precisely because there's nothing out there that does this well and is easy to use.

Coda, AFS, InterMezzo

[RAMMS+EIN]

There have been some efforts in the area of networked filesystems with disconnected operations. I remember checking out AFS [wikipedia.org], Coda [cmu.edu], and InterMezzo [inter-mezzo.org] years ago. At the time, I found something wrong with each of them, but they may have improved since then. Of the three, I think Coda is your best bet.

you already solved your problem

[coaxial]

Simply instally unison or rsync or whatever and have the job kick off with whereami [debian.org] for linux (you'll have to find the main page yourself) or marco polo [symonds.id.au] for macs.

Re:rsync

[syphax]

Unison [upenn.edu] is 2-way rsync. But as the poster noted, unison/rsync doesn't easily support automatic synching (that I know of)- you have to kick it off and then deal with any conflicts, etc., manually. I think the poster is looking for ideas of at least automating Unison/rsync (BTW does rsync support 2-way updating, as the poster explicitly mentions?).

As someone who relies on running unison manually (too lazy to figure out how to automate on my Windows box), I'd be interested in relevant solutions.

Re:Windows might be good for something

[Anonymous Coward]

I've yet too seen it done well in Win2K/XP as well. I've had tons of issues having My Documents on a server and making a offline copy. Windows would during copying of files just drop the connection and go into offline mode etc. It's a great idea but I have yet to see it work reliably.

Re:rsync

[EvanED]

The information on how to use this hack is not publicly available.

Um... what?

You mean besides this diagram [microsoft.com] of the steps you should follow when making a backup (and a similar one for restore), and the MSDN documentation [microsoft.com] for the VSS.

Re:Windows might be good for something

[arivanov]

Offline most likely derives its origins from Coda which was designed to work for 100MB at most. It seems to inherit all of its problems when the data volumes become big. I have had to support an environment where people casually offlined 3-4GB documentation trees and it was falling over on regular basis.

Further to this, offline files has a number of fairly fundamental bugs in the actual implementation. It records both the IP and the name of the server somewhere when doing the offlining. As a result if the name (but not the drive) or the IP changes your entire offline tree goes south and stays offline. You can neither delete it nor reconnect it and the only way of dealing with this is either surgery to the network (aliasing IP addresses) until you reconnect. The only alternative is to rebuild the affected laptops from scratch.

SyncBackSE

[__aalmrb3802]

If you're running Windows, I would recommend SyncBackSE (http://www.2brightsparks.com/syncback/sbse.html [2brightsparks.com]), which I expect you should be able to setup to do exactly what you asked.

OpenAFS

[sid77]

As said before there're many choices, each ones with its own pros and cons, so I'll throw this one in: OpenAFS [openafs.org].

As read from the main page:

What is AFS?

AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for federated file sharing and replicated read-only content distribution, providing location independence, scalability, security, and transparent migration capabilities. AFS is available for a broad range of heterogeneous systems including UNIX, Linux, MacOS X, and Microsoft Windows

Hope this helps, ciao

Re:iFolder?

[hendersj]

iFolder3 lets you specify whatever folder you want - I sync about 5 or 6 folders and share them with different people in my department.

(And if you're who I believe you are (CC), hey! Drop me a line...)

Re:Windows might be good for something

[Anonymous Coward]

Periodic connection drops are almost always due to samba. It just isn't robust enough to use for something like this -- if you hammer the smb part, the nmb part won't give timely replies (or was it the other way around?), and the netbios share will be marked as offline.
It can sometimes be alleviated by ensuring that a Windows box is the master browser, by setting "local master = No" and "preferred master = No" in smb.conf.
Alternatively, if you only have W2k/XP clients on the network, "disable netbios" seems to do the trick too.

Re:common refrain

[oatworm]

Not that it matters, but since you asked...

Photoshop -> GIMP [gimp.org]
Avid -> LIVES [sourceforge.net] - Note: I am not a video editor and have no idea if this program is any good.
Quicken -> GNUCash [gnucash.org], among others.

I guess what I'm saying is that, based on your definition of "silly", there's quite a bit of silliness going on in the world today. *grin*

Re:rsync

[dusty123]

I don't see how rsync solves this problem:

AFAIK, rsync is only one-way, meaning that it overwrites and eventually deletes files. Have a try:

mkdir d1 d2 # Create two directories (e.g. one on server, one on laptop)
touch d1/foo.txt # Create an empty file
rsync -r d1/ d2/ # Sync the directories
echo "123" > d2/foo.txt # Now modify the file on d2 (e.g. laptop)
rsync -r d1/ d2/ # Sync again
cat d2/foo.txt # Ooops - foo.txt is empty!

One possible way I experimented with is the following:

- Integrate a rsync server -> laptop in the startup procedure of the laptop
- Never modify a file on the server while working with the laptop
- Integrate a rsync laptop -> server in the shutdown procedure of the laptop

In theory this works, but practically there are cases where you miss the shutdown/startup sync, e.g. when you have no network at startup (e.g. you took your laptop away from home and forgot to sync it), in case you laptop crashes, the network fails during shutdown and numerous other problems. These lead to dangerous situations, e.g. if the rsync laptop->server fails during shutdown, a startup-rsync may overwrite modified files.

After loosing some of my work, I decided to switch to unison, which is a 2-way sync and lets me decide how to resolve syncing problems.

Nevertheless I'm not entirely happy with the situation - if I forget to sync, I have to resolve things manually, moreover the sync takes quite some time.

In my special case, I have a WLAN connection to my server most of the time, so changes could be written immediately. So I'd favour some kind of network file system that has offline capabilities and can handle two-side modifications in some way. I thought about Coda but it seems to be far too complicated and unreliable and I don't know better alternatives.

So I'm still stuck to my Unison solution, which is somehow cumbersome, but works...

Unison, Rsync & NTP

[Colin Smith]

Unison can be scripted, added to a login script. As can rsync on windows. Alternatively you can add a polling batch file which wakes up every so often and checks to see if the server lives. (Yes, even on Windows)

Rsync can sync in both directions, but you decide one of the sides is the master and sync that one first, in the case of conflicts the master rules. It isn't possible to choose on a file by file basis at sync time as you can with Unison.

Oh, and NTP is absolutely vital when doing any synchronisation.

Basically. Either you do it manually and manage conflicts at sync time, or you do it automatically and define one of the sides as a master in the case of conflict. There's really no way round this, software just isn't sophisticated enough to decide what you're thinking.

The truth is that filesystem syncing isn't ideal for a very dynamically updated file system. It is best used on fairly static filesystems or one way syncing. Documentation, backups and the like.
 

Re:Windows might be good for something

[dos]

It records both the IP and the name of the server somewhere when doing the offlining. As a result if the name (but not the drive) or the IP changes your entire offline tree goes south and stays offline.

Go download csccmd 1.1 from Microsoft.

csccmd /moveshare will take care of this.

Re:rsync

[Znork]

"AFAIK, rsync is only one-way, meaning that it overwrites and eventually deletes files."

rsync has a whole bunch of options that will let you decide behaviour. --update will make it skip files that have newer modify times or you could use --backup to make it make a copy of files instead of overwriting them, etc. Mix and match and run two-way syncs after eachother and you could get close in behaviour to a real two-way sync.

"I thought about Coda but it seems to be far too complicated and unreliable and I don't know better alternatives."

I've played around with Coda, and from what I recall there are two things that make it impractial for 'ordinary' use. The lack of file locking (which causes problems with annoying apps that use it) and the handling of large files (it had to copy the entire file to local cache before unblocking the io calls, ie, dont look at any video files on coda). And so, my original idea of having home directories supporting disconnected operations were shot. It would have worked very well for specific subsets of datastorage, but in the end it was simpler to just sort the data into various structures and deal with syncing on a case by case basis (rsync for some things, plain nfs/autofs for other things, cvs for code or text, etc).

In the end, I think this is one of those problems where it's better to just sit on your arse and wait because the problem of permanent connectivity will be solved before someone figures out how to make a wholly transparent redundant filesystem that seamlessly supports disconnected operation. The whole problem is simply to a certain extent incompatible with the way filesystems usually work.

Re:Coda

[wildjim]

Funnily enough, I've been working on exactly this over the weekend. Unfortunately, the docs are terrible.
Some of the Wiki info suggests that things have improved, but I'm discovering a decent-sized client cache (10Gb) so that I can offline most of what I'd use has horrendous occaisional slow-downs and pauses.
I'm planning on testing a local/client server and a client with the RVM turned-off this week, but I'm not keen on the size that the RVM file(s) will have to get to.
Previous comments in here suggesting it's not really designed for modern data-sets (gigs rather than megs) are starting to look as if that's true... Other than that, it actually looks like a reasonably good design!