Laptop/Server Data Synchronization? 305
gbr writes "I've been trying to automatically synchronize data between a laptop and a server. When the laptop is connected to the network, I want all writes to automatically propagate across to the server. When the laptop is disconnected I want the laptop user to continue working with the local data. When the laptop is reconnected, I want the data to automatically re-sync.
The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop. The data doesn't contain anything too special, no database tables etc. It does contain binary data such as executables and word processing documents. I've looked at ChironFS, Unison file sync, and drbd. ChironFS needs a manual rebuild if a connection fails, and the user needs to know which machine contains the correct data. Unison requires the user to initiate the synchronization process manually every time, and drbd is just not meant for the job at hand. How do you automatically, and invisibly to the user (except in the case of conflicts), synchronize between a laptop and a server?"
rsync (Score:5, Informative)
Re:rsync (Score:5, Informative)
I'd also take a look at Microsoft's SyncToy [microsoft.com] if you're on win***s.
Re: (Score:2)
Re: (Score:2)
Windows might be good for something (Score:4, Interesting)
My Briefcase in Windows 95. It even has a cute ickle briefcase icon.
Somewhat seriously, Offline Files in Win2K/XP is something I've yet to see done well on any other OS.
Re:Windows might be good for something (Score:5, Informative)
Further to this, offline files has a number of fairly fundamental bugs in the actual implementation. It records both the IP and the name of the server somewhere when doing the offlining. As a result if the name (but not the drive) or the IP changes your entire offline tree goes south and stays offline. You can neither delete it nor reconnect it and the only way of dealing with this is either surgery to the network (aliasing IP addresses) until you reconnect. The only alternative is to rebuild the affected laptops from scratch.
Re:Windows might be good for something (Score:4, Informative)
csccmd
Re: (Score:2, Informative)
Re:rsync (Score:5, Insightful)
Personally, I play with bzr most frequently; it has a nifty Python interface (and an extensive plugin architecture) which makes it quite conducive to local scriptage. (As an example -- I have a local, filesystem-backed set of CA scripts which use bzr for transactional semantics; if a method is called which throws an exception, all filesystem changes are automatically rolled back; if a method succeeds, a commit is done to record the operation and [effectively] set a rollback point). The separation between working tree and repository is optional (by default, all working trees are also repositories -- much like BitKeeper in that respect), which makes it very handy for situations like this where you don't necessarily *have* a separate, central location which all nodes can always communicate with, and where the different trees are allowed and expected to temporarily diverge.
Subversion (Score:2)
I agree—subversion isn't really the right tool for syncing data in two places. Having said that, if you want to keep revision history and sync it, then the latest subversion might be right up your alley. Included is a svnsync tool that is intended to sync a "live" repo (one that you use regularly for commits) with a backup repo (exists solely as a copy).
As far as automating backup, assuming you're talking about linux, I would say that the best approach is to simply write a short script that pings f
Re:Subversion is for stupid and ugly people (Score:4, Insightful)
Out here in the Real World, folks setting up revision control systems need to count "stupid people" (read: artists and web designers who are too busy making art or designing web pages to care about revision control except inasmuch as it's a way to back up and distribute their work) among their customers. For a great many cases, subversion is Good Enough, and it has excellent Windows support, integration with just about every IDE under the sun, TortoiseSVN and other nice pretty hand-holdy tools available which simply aren't ubiquitous among SCMs written with hardcore users in mind (seemingly to the exclusion of those that aren't). SVN isn't distributed, which sucks. SVN has an ugly hack of an excuse for a rename handling algorithm, which sucks. SVN is slow as molasses compared to some of its competition and lacks merge tracking (and thus history-sensitive merging) and has a gawdawful working tree library and sucks in any number of other ways -- but it is a compelling replacement to CVS, and sometimes that's what the customer needs, no matter what shiny happy features ${YOUR_FAVORITE_SCM} may have and no matter how many ways SVN manages to annoy the power user.
And trust me, I learned this one the hard way.
As for the 4NT copy suggestion, the whole bidirectionality and rename handling arguments come into play.
Re: (Score:2, Informative)
Re: (Score:2)
Re:rsync (Score:5, Informative)
By using pooling and compression, one client of mine is using BackupPC to backup over 1TB of data distributed among over 100 laptops to a 200GB filesystem on a central server. The network is polled every hour, and any system that hasn't been backed up in the last 24 hours is queued. Beautiful system.
Re: (Score:2)
Re: (Score:2, Informative)
Um... what?
You mean besides this diagram [microsoft.com] of the steps you should follow when making a backup (and a similar one for restore), and the MSDN documentation [microsoft.com] for the VSS.
Re: (Score:2)
Re: (Score:3, Interesting)
As far as the user is concerned his primary concern for laptop data loss is email. So you have to back it up as a part of any backup solution. If you are storing email locally on the laptop and backing it up the backup will nearly alw
Re: (Score:2)
Re: (Score:2)
The same goes for many other files that have running services locking them. Just pop in a pre- and post-backup script in BackupPC that handles stopping and starting said services or have the services (i. e. RDBMS) do a nightly dump to a file.
The beauty of BackupPC is that it will even only trans
Can BackupPC be used with Cygwin on Windows? (Score:2)
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:3, Funny)
Re:rsync (Score:5, Informative)
Re:rsync (Score:5, Interesting)
- syncs continuously/watches the FS for file changes (no cron jobs needed -- things usually sync as quickly as they can be sent)
- does binary diffing and only sends deltas (compressed & over SSL)
- transparently archives past versions of all files (i.e. undelete/infinite undo)
- syncs across any number of machines
- lets you get to your files from the web
- some more info @ http://venturebeat.com/2007/08/16/the-y-combinato
We made it after hacking together our own rsync-based abominations and getting really annoyed that no one had solved this genre of problems in a way that normal people could use.
Okay, I can stop shilling now. I was just excited that other people run into these problems.
Re: (Score:3, Interesting)
This looks quite similar to Novell's iFolder but with you running the server yourselves instead of having your users set up an iFolder server. Last I used iFolder was in the 1.x or 2.x days and it frankly wasn't anywhere near the polished product you have here. Now it seems that iFolder 3.x [ifolder.com] is open source and looks a lot more polished.
Still, I think you surely have a great service market here even though the polished front-end app seems to now be done open source. Best of luck to you on your new ventur
Re: (Score:2)
Re:rsync (Score:4, Informative)
Unison [upenn.edu] is 2-way rsync. But as the poster noted, unison/rsync doesn't easily support automatic synching (that I know of)- you have to kick it off and then deal with any conflicts, etc., manually. I think the poster is looking for ideas of at least automating Unison/rsync (BTW does rsync support 2-way updating, as the poster explicitly mentions?).
As someone who relies on running unison manually (too lazy to figure out how to automate on my Windows box), I'd be interested in relevant solutions.
Unison, Rsync & NTP (Score:4, Informative)
Rsync can sync in both directions, but you decide one of the sides is the master and sync that one first, in the case of conflicts the master rules. It isn't possible to choose on a file by file basis at sync time as you can with Unison.
Oh, and NTP is absolutely vital when doing any synchronisation.
Basically. Either you do it manually and manage conflicts at sync time, or you do it automatically and define one of the sides as a master in the case of conflict. There's really no way round this, software just isn't sophisticated enough to decide what you're thinking.
The truth is that filesystem syncing isn't ideal for a very dynamically updated file system. It is best used on fairly static filesystems or one way syncing. Documentation, backups and the like.
Re: (Score:3, Interesting)
Re:Unison, Rsync & NTP (Score:5, Insightful)
Re: (Score:3, Informative)
AFAIK, rsync is only one-way, meaning that it overwrites and eventually deletes files. Have a try:
mkdir d1 d2 # Create two directories (e.g. one on server, one on laptop)
touch d1/foo.txt # Create an empty file
rsync -r d1/ d2/ # Sync the directories
echo "123" > d2/foo.txt # Now modify the file on d2 (e.g. laptop)
rsync -r d1/ d2/ # Sync again
cat d2/foo.txt # Ooops - foo.txt is empty!
One possible way I experimented with is the following:
- Integrate a rsync server -
Re: (Score:3, Informative)
rsync has a whole bunch of options that will let you decide behaviour. --update will make it skip files that have newer modify times or you could use --backup to make it make a copy of files instead of overwriting them, etc. Mix and match and run two-way syncs after eachother and you could get close in behaviour to a real two-way sync.
"I thought about Coda but it seems to be far too complicated and unreliable and I don't
Subversion (Score:2, Informative)
Re:Subversion (Score:5, Interesting)
At the risk of saying something stupid or blasphemous: why offer something that requires "writing some scripts"?
If the OP wanted to "write some scripts" s/h/it could have done all the neccessary work with a couple foreach...cp...end. Or, hey, rsync.
I am suspecting the OP is wondering whether there isn't something out there that "just kinda works" and only needs intervvention in case of a conflict.
Knowing well that this will definitely be considered blasphemy: I've been using Window's "briefcase" system since Win98. It does "kinda work". Most of the time. And requires work when there's a conflict. Which appears to be what the OP is looking for. Given that the OP doesn't seem to want to just go that route, the question appears pertinent what s/h/it is looking for that Mr. Gates briefcases can't/won't do...
Re: (Score:2)
No, you have a good point.
What I meant by ``writing scripts'', though, was like a line in xinitrc that launches ``svn up'' on startup, and SSH keys if he doesn't have them already. If he wants, also a cron job every how-ever long to run ``svn up''
Re:Subversion (Score:5, Funny)
You may want to reevaluate your approach to political correctness.
Re: (Score:2)
Re: (Score:3, Funny)
iFolder? (Score:4, Informative)
http://www.ifolder.com/index.php/Home [ifolder.com]
Re: (Score:2, Informative)
Re: (Score:2, Informative)
One possible problem: you have to store the information in a folder (which you specify). Only the data in that folder is synced.
HTH
-Fr
Re: (Score:3, Informative)
(And if you're who I believe you are (CC), hey! Drop me a line...)
Re: (Score:2)
Re:iFolder? (Score:5, Interesting)
I don't know how mature the novell version is but the open source version is very far from being mature. In fact there hasn't been a stable release in more then two years and nobody knows if or when there will ever be a stable release.
A while ago all the developers on ifolder either quit or were fired and the development was moved to india. Since then the pace of development has slowed down to a crawl and the new developers try to understand the code base and fix bug reports.
Right now you can download something that is beta-ish but I certainly would not trust my mission critical data to it. If you want something that works you are going to have to pull off the trunk and compile it.
Unison (Score:2, Informative)
my take (Score:5, Insightful)
I can't give you a flat out solution, because all situations are different. But I can pass on a bit of wisdom. The most important thing for you to do is create business rules for your synchronization. If the data on the server has changed and you made changes offline, who gets priority? You will have three categories of which a file can be... Client changes get priority, Server changes get priority, and Merge files. I would stay away from the last one. If you want to keep things simple, Id go for the "Server changes get priority" approach. In short, if you took an "online" file "offline" and came back, and the server copy has changed since, your offline edits are abandoned. This way, it makes it so heavily edited files have a shorter "check out window" (even if you don't use a checkout system), and forces the person taking the file offline to coordinate with everyone else that may edit this file.
common problem (Score:2, Insightful)
And there you have the problem with synchronization. There's no mind reader program (yet) so sorry but you're going to have to make up your mind about how to handle it when the server version changed too. Either find a way to merge the files or start making decisions about when they can get modified (ie a checkout system) or i
Re: (Score:2)
Since the original questions said that the sync should work "invisibly to the user (except in the case of conflicts)", I'd guess they have given that some consideration. Besides, I can easily imagine a piece of software that is configurable to do any of the various options you suggested.
Re: (Score:2)
Man, the mods are definitely asleep today.
LS
common refrain (Score:2)
This is a common refrain, and I must say I find it puzzling. What about Photoshop for photo editing? Or Avid for video editing? Or Quicken for accounting? Where are the freeware versions of those applications? MS office applications happen to have free alternatives in OpenOffice and NeoOffice and so on, but only as a result of monumental interest in staging a pushb
Re: (Score:3, Informative)
Photoshop -> GIMP [gimp.org]
Avid -> LIVES [sourceforge.net] - Note: I am not a video editor and have no idea if this program is any good.
Quicken -> GNUCash [gnucash.org], among others.
I guess what I'm saying is that, based on your definition of "silly", there's quite a bit of silliness going on in the world today. *grin*
Am I missing something? (Score:3, Insightful)
"The issue is, the data on the server may have changed as well, which needs to propagate back to the laptop."
So let me get this straight.. You have the old version of the file, somewhere. The new laptop version of file, somewhere. And the new server version of the file, somewhere. And you want the software to decide which to use and copy it to both the server and the laptop?
There are even more issues here, but it kinda sounds like you want some artificial intelligence that you can download.
Aero
Unison (Score:2, Informative)
You might have to do A-B, A-C, A-B type syncs for more than 2 paths, unless you stick to a hub/spoke or cascading distribution model.
Not all conflicts are automatically resolved, by default.
http://www.cis.upenn.edu/~bcpierce/unison/ [upenn.edu]
Good luck.
Re: (Score:2)
It's not that complicated. You have a directory structure on machine A with files and folders. On machine B, you have - wait for it - the same structure. SO synching is a matter of comparing file lists, hashes, last-modified timestamps, and last-synched timestamps. It only gets sticky when both files have changed since the last sync, which is an exception that usually merits human review to sort out which one to keep. If you have reasonable practices that minimize collisions, no big deal. Of course, i
Yes, you're missing something! (Score:2)
The usual way of handling this is by comparing the timestamps on the directory too, and the one with the newer timestamp wins. This helps prevent new files from being deleted, but it isn't safe. Cons
Re: (Score:2)
The entire sync/remote fs approach is fundamentally wrong for any place that has more than 2 people working on the same project/material.
What if you DO NOT WANT to resolve the conflict right away? What if the document that has been changed by two people is 200 pages long and the changes are substantial and you have 10 minutes to get the data and run? Even if you have the time there are plenty of other reasons why you may want to postpone the merge and review.
The only real solution is to keep documents i
Re: (Score:3, Informative)
Not seeing the difference between arbitrary files on a disk and files that have been explicitly version controlled is I guess what makes you the hardware guy - does that mean you nail the floorboards down?
Re: (Score:2)
So, if I make changes to function A and you make changes to function B, the server says "OK, they changed different parts, so I'll apply person A's change, then person B's change creating version C."
This is overly simplistic, but you might get the idea.
Re: (Score:2)
Person A and Person B both check out file Version 1.
Person A modifies the first half of the file.
Person B modifies the last third of the file.
It comes time to merge. Typically, one of them commits their changes first. For example, Person A commits. Now the file is version 2.
Now, Person B attempts to commit, and is told that their version is obsolete. They ask for an update.
The up
Not easy at all (Score:2)
Or, you could use a versioning system, making a new point-series branch whenever there was a difference between laptop and server versions. But you would still be left with the problem of choosing which edits to save and which to dismiss.
In the end, you need humans to discuss the changes made and assimilate them into a new master document whenever there is a differ
Re: (Score:2)
Yep. And it magically works even in the face of conflicting changes, and never breaks. Others have been struggling with this for years and years, but now Microsoft has come and saved the day: they've implemented their own system, different from all the others, and it's better and more user-friendly. As always.
(And yes, I was being sarcastic)
Unison and a cron job or login script. (Score:2)
Coda (Score:5, Insightful)
Re: (Score:3, Interesting)
I've always wanted Coda to work for this, but I haven't ever gotten it working. My current thinking is that I'd need to set up a server on my laptop, and then have the client talk to the local server. Then the two servers could sync.
Does anyone have any information or case studies on how to make this work for a small network? Easy conversion tools? It seems like the ideal solution to me, but getting it to work seems difficult.
--Ted
Re: (Score:2, Interesting)
I haven't actually used Coda, though I'm planning on it for a small network myself, because this is exactly what it was designed to do. But why would you need the server on the laptop? All you should need is a client. Have you tried it that way and it didn't work or something?
I do hear it's a pain to set up though.
Windows & Make Available Offline (Score:5, Informative)
In Windows you can mark a folder on a network share as "Available Offline". Windows will copy all of the files to the local HD and if the server isn't available just work with the local copies. When the server is detected Windows will automatically sync the files and pop-up asking the user about conflicts (keep local / keep remote). When connected writes automatically go to both the local copy and the server.
One of the few places that Windows has right and I haven't found a Linux or OS X solution for that is nearly as nice.
Re:Windows & Make Available Offline (Score:5, Informative)
Re: (Score:2)
Care to elaborate how did it broke up and how one can recover from such a situation?
Re: (Score:2)
But for it to 'continually' break or have problems as you suggest it would more likely be a situation where the infrastructure was not setup properly in the first place, with poor shares designs, trying to use Samba servers, etc...
If properly setup, even in high volume business environments we have dealt with clients with 1,000s of users all utilitizing offline files and even offline profiles (including desktops, etc), and the fail rate is virtually 0.
Also
Re: (Score:2)
Re: (Score:2, Interesting)
I use it with my wife for our financial data. She syncs, makes changes, then next time I connect I can choose to use the server version or my version. Since there is no version control, you have to communicate. Then again, with the way version control works, if you end up merging a lot, perhaps too many people are working on the same problem anyhow.
In the e
Re: (Score:2)
Foldershare (Score:2, Informative)
Re: (Score:2)
The only drawback is that the maximum number of files allowed per synced share is 10k. While this sounds like a lot, it really is not.
OS X Client & Server does a good job with this (Score:4, Insightful)
When you have a mobile user account (i.e. a network account with a local copy of the home folder on the workstation), it will sync every so often (frequency and exactly what is synced/skipped can be configured on the server end, and the user can kick it off manually from the client end). To the best of my knowledge, the sync is bidirectional, so if you log into another machine with a mobile account and modify the server copy, the changes will be reflected on the mobile copy at next sync. It makes my life easier because if a laptop user's machine gets lost, stolen, damaged, or destroyed, we've automatically got a backup copy of the data on it up to the last time it was synced.
In the event of conflicts, the user is presented with a dialog asking which version to keep, including file size and modification date.
Note that I'm not suggesting you throw out your existing hardware and buy Macs to get this feature, but maybe look into exactly how it's done on the Mac and see if you can duplicate it on your systems.
~Philly
In OSX, portable home directories (Score:5, Informative)
Try coda (Score:2, Interesting)
Collaborative software? (Score:2)
commercial products. (Score:4, Interesting)
Novell's iFolder is a very interesting alternative... runs on Linux/Apache/Java stack & only transmits changed blocks over an SSL connection.
Other things worth looking into include Microsoft Groove--let's you synchronize an entire workspace with yourself on other computers or other people - and is relatively network & environment-independent (though Windows only)
Synchronization Woes (Score:5, Interesting)
The more complex problem has been thrown at me a few times. What if it's not just one person?
Say you have a repository of data that a dozen people may be working in. When they're all network connected, they're all dealing with the same file pool. When they take their off-line copies with them (unplugged laptops on vacation), they all make changes to the same files. Maybe mine is a one line change. Maybe one guy copy&pasted the first 3 chapters from War And Peace into a comment somewhere in the middle. Maybe another developer did some very intellectual looking changes but hosed some major functionality.
When you start putting machines back on the network, who is right? The 6 guys who did real work are obviously right(ish), but they all made different changes. The very last change will end up being someone's 3 year old kid who was pounding on the keyboard right before daddy shut down the laptop, saving the new changes. Probably the last is the most recent, and right by most methods.
It's not a pretty picture, and requires some intelligence to sort out the mess.
The only "good" resolution I've found is to give logical authority to the changes. Bob is in charge of development. Any changes going into the development or production tree must clear him. He should be able to recognize that the 6 guys made changes, and diff them to come up with the common changes. The 3 chapters of war and peace go by the way side. And the guy with the 3 year old "developer" gets reprimanded.
In the end, a good revision system and good backups are needed too. Something will slip through the cracks, and you'll need to roll back to something you hope is good.
I take control over whatever I'm working on, so if I know I'll be working offline, I'll scp the data to my laptop, work on it on the road, and scp my changes up to the server when I'm done. Anyone else who may have worked in my project space in the duration should have known better.
Re: (Score:2)
Coda, AFS, InterMezzo (Score:4, Informative)
Synctoy (Score:2)
Handy Windows Syncing Tool (Score:2)
PathSync [cockos.com] by Cockos (Justin Frankel of Winamp fame's new company).
It has some automation features as well; haven't used them so can't vouch for them.
you already solved your problem (Score:3, Informative)
iDisk does this (Score:2)
SyncBackSE (Score:3, Informative)
OpenAFS (Score:3, Informative)
As read from the main page:
AFS is a distributed filesystem product, pioneered at Carnegie Mellon University and supported and developed as a product by Transarc Corporation (now IBM Pittsburgh Labs). It offers a client-server architecture for federated file sharing and replicated read-only content distribution, providing location independence, scalability, security, and transparent migration capabilities. AFS is available for a broad range of heterogeneous systems including UNIX, Linux, MacOS X, and Microsoft Windows
Hope this helps, ciao
In great *nix tradition (Score:2)
Use something like http://0pointer.de/lennart/projects/ifplugd/ [0pointer.de] to launch a script that runs Unison file sync when ever you are connected to the network.
CacheFS - on OpenSolaris (Score:2)
You can even prime the cache using cachefspack initially. It works in disconnected and connected mode and is automatic.
An alternative tool on OpenSolaris distributions is filesync, it uses the same config syntax as cachefspack does but works by simply using
cp/mv/rm/chmod/chown and doesn't include its own transport layer (so needs to use NFS or similar).
inotify... (Score:2)
Don't automate it (Score:2)
You probably should make this manual (at least via a confirmation dialog), so that if the user is only connecting for a few seconds, it does not try to sync. Also, the user can then control priority - he may want to get to the web first, rather than waiting while an Office service pack downloads...
Conduit? (Score:2)
It does however look like a very ambitious project aimed at syncing both files and other data between both computers and other services.
Windows Vista (Score:2)
Roaming profiles (Score:2)
I have the kerberos single sign on for a multitude of services. I have the LDAP user metadata. I have NFSv4 file shares secured with kerberos, and autofs homes that enable me to use the same app settings on each machine. I have a groupware server (Kolab) serving all of my users (all two of them, ha!). But can I find any way of accessing my home drive (reliably) on my laptop when it's not plugged into the network?
If
CVS (Score:2)
I guess any other version management tool can be used too, I just tend to know and like CVS.
- Hubert
Re: (Score:2)
You could also look at getting a 2.5" portable USB or Firewire drive and storing the VM's on there. While it would be an extra thing to car
Re: (Score:2)
is there anyone here with Groove experience? I'd love to hear whether