Hot Comments
- About time! (5 points, Informative) by sackvillian on Tuesday February 09, @09:50AM attached to Call For Scientific Research Code To Be Released
- Re:Be careful using the P2V tool. (5 points, Informative) by klocwerk on Tuesday February 09, @03:41PM attached to The Hidden Treasures of Sysinternals
- W.A.G. (5 points, Insightful) by ipb on Tuesday February 09, @01:17PM attached to How Do You Accurately Estimate Programming Time?
- Re:Surprise (5 points, Insightful) by Anonymous Coward on Tuesday February 09, @04:38PM attached to Microsoft Says Windows 7 Not Killing Batteries
- No problem (5 points, Insightful) by rastoboy29 on Wednesday February 10, @05:27AM attached to Appeals Court Rules On Internet Obscenity Standards
7131106
story
Comments: 305 +- Ask Slashdot: Synchronize Data Between Linux, OS X, and Windows? on Thursday November 19, @04:30PM
background: url(//a.fsdn.com/sd/topics/topicstorage.gif); width:48px; height:51px;
storage
background: url(//a.fsdn.com/sd/topics/topicapdesktops.gif); width:43px; height:73px;
mac
background: url(//a.fsdn.com/sd/topics/topicmediaall.gif); width:73px; height:59px;
media
background: url(//a.fsdn.com/sd/topics/topicsoftware.gif); width:65px; height:65px;
software
aaaaaaargh! writes "I'm using a laptop with Ubuntu 8.04 for work, a netbook with Ubuntu 9.10 when I'm outside, Mac OS X 10.5 for hobby projects, and Windows XP for gaming. For backups, I'm currently using Jungle Disk and Apple's Time Machine, and I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another. Options range from putting everything online (but Jungle Disk disks seem to be too slow for anything else than backup), storing my data on external media like USB sticks or SD cards, or working with copies by synchronizing folders over the network. I don't want to give my data away to some server outside without strong encryption (controlled by me, including the source code) and external media like USB sticks are a bit too fragile according to my taste. The solution should be reliable, relatively failsafe, as simple as possible, and allow me to continue to use Jungle Disk for backup. So what would you recommend?"
Related Stories
A great many people think they are thinking when they are merely
rearranging their prejudices.
-- William James
All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.
Rsync? (Score:5, Informative)
Set up one computer as a server and rsync/ssh to it with either a cron job or at your whim.
Re:Rsync? (Score:5, Informative)
ZFS can also do compression, and automatic snapshots, not to mention rock solid data integrity with raidz2
Seriously what more do you want from a storage/backup solution ?
Parent
Re:Rsync? (Score:4, Informative)
Better yet, Setup ZFS on a opensolaris box, export the ZFS volume as NFS/Samba share or as an iSCSI target, access from Linux,Solaris,Windows etc.
Note that newer releases of OpenSolaris have built-in kernel support for CIFS with ZFS.
# zfs set sharesmb=on mypool/mydata
# zfs set sharenfs=on mypool/mydata
Mapping between Unix UIDs and Windows SIDs is done transparently, and you can change things from the Windows GUI if desired and the ACLs will go live on the Unix side as well: SIDs are first class security objects. None of the usual Samba UID/SID kludginess (though (Open)Solaris comes with a Samba package if you want to go that route.)
Case-(in)sensitivity is also a user toggle away if so desired.
De-dupe was also recently committed to the code base. Crypto is being worked on and I think patches are available.
Parent
Re: (Score:3, Informative)
You're going to have some file semantics issues in any case. Those introduced by CIFS (don't see how ZFS is a factor) are pretty easy to deal with.
Re: (Score:3, Insightful)
BTW, from my perspective your own post is more typical of Slashdot than my own: theorizing without data, postulating ignorance on the part of those with experience, and generally trying to adopt a tone of undue authority.
Re:Rsync? (Score:4, Interesting)
Except it didn't exactly answer his question. He appears to favor local storage. Ha already has a backup solution and if he wanted a snetwork share, I'm pretty sure anyone running the 3 major OSes can figure out how to share a drive via NFS/AFP/SMB.
For a home user, ZFS's big achilles is the inability to change raidz geometry. Most home users add another drive to increase capacity, *not* another raidz of drives into a pool. Nor do they typically want to replace all their raidz drives with larger drives. ZFS fails on that particular common home storage scenario. Everyone knows it and it's been talked about to death. And btrfs is looking better every day.
The answer he's looking for is probably some form of rsync or unison solution. Depends on the complexity of his syncing. And if he needs/wants some form of version control. He knows how to use SVN so he's probably not needing version control.
I'm curious too; I have to keep local assets for CGI apps in sync on a few different systems and rsync is the best I've come up with. You can't drop extra files into the asset directory structure and you can't rely on their state to tell you anything about the need to sync.
Parent
Re: (Score:3, Insightful)
None of those protocols were designed to work well over the public internet. Sure, NFS/SMB/iSCSI are IP protocols but they don't work well over high latency and low bandwidth connections.
I'm not arguing with your choice of ZFS; I would just look at something more simple like rsync over ssh to a ZFS volume instead of sharing NFS/CIFS/SMB/iSCSI out to the Internet.
Re: (Score:3, Informative)
Better yet, use Mercurial (or any other distributed version control system). It is designed for this kind of use!
I have three clones (of a project), one in desktop, one in laptop and one in an USB stick. Just "pull" (or "push") and "commit". Sometimes a "merge". With network connectivity you do not need the USB stick (in my case it is just a bit easier).
Should any of those clones/repositories die it would not be a big problem.
Note: this is not a backup solution, this is a solution to sync the machines!
Re:Rsync? (Score:4, Informative)
luckybackup is a gui for rsync and allows you to sync folders.
I use it to quickly backup everything. It doesn't allow versioning, but I use squashfs-tools to remove all redundant files and compress them with lzma. So I have a folder system of dates for versioning in a highly compressed squashfs file system container, that's mountable.
sudo apt-get install luckybackup squashfs-tools
Parent
Re: (Score:3, Informative)
of course you can. the ssh support was added on the top of rsync. It originally used the rsyncd daemon which I believe do not use encryption. (similar to rlogin for authentification)
Re:Rsync? (Score:5, Insightful)
Don't worry about the encryption overhead, just make sure AES is the first cypher chosen. AES is VERY fast.
Parent
Unison (Score:5, Informative)
I like Unison [upenn.edu] for this sort of thing.
Re:Unison (Score:5, Informative)
I second the suggestion. It's basically like rsync except it's more like real synchronization. What I mean is, AFAIK with rsync you basically have to say, "Sync folder A and folder B by assuming that folder A is the copy I want to keep, and changing only folder B." With Unison, on the other hand, you can say, "Sync folders A and B by keeping track of when files get modified, and keeping the most recent version of each file. If a file has been modified in both folders since my last sync, let me know and ask me what to do."
One minor annoyance is the project page links to an OSX version that includes a Cocoa interface, and that's kind of a pain if you want to script it to run in the background. Even if you call it from the command line, the GUI will still open up. Also, if you install it from MacPorts, it doesn't do a good job of maintaining resource forks. Of course, if you're syncing to Linux or Windows, you might not care about resource forks.
I think the best place to get the OSX version is here: link [haifa.ac.il]
It's been a while since I've downloaded it, though.
Parent
Re: (Score:3, Informative)
The Cocoa interface can be disabled if you're writing a script to run in the background. There is a "-ui text" flag to the command line tool (/usr/local/bin/unison) that will prevent the Cocoa GUI from being show
Re: (Score:3, Insightful)
So why bother? Why not use distributed VCS?
They are quite easy (to learn), they handle merges apparently much better, they give version history, rollback, branches, etc. After all, the OP was using SVN.
Re:Unison (Score:4, Informative)
I cannot say whether unison is the best solution, but it is one that I have been using for 7 years or more, between two desktops (home and work), a laptop an external hard drive and another offsite backup. These were on a variety of filesystems between Linux and Windows flavours. MacOS X should be no different.
While I have seen graphical interfaces to unison, the command line interface is so much easier to use.
I would definitely recommend unison... while it is not a revision control system it is an effective solution at working on the same set of files wherever you are provided you synchronise twice a day.
Parent
Re: (Score:3, Insightful)
Agreed - but if rsync works fits, it is preferable.
That said, the constraints where rsync works flawlessly are pretty strict:
1. You always rsync down at the start of a session
2. You always rsync up at the end of a session
3. You never have more than 1 session
Just make sure you use the best settings - don't forget --delete or whatever it is to handle removed directories.
Unison works much better due to its 2-way change propogation, but it is only designed to handle 2 sources of documents, not 3.
I've never had
Re:Unison (Score:5, Informative)
I agree, unison is great. I used unison for several years to sync machines. It worked great for linux-linux, and seemed to mostly work for linux-mac/mac-linux syncs as well. The main problem was with resource forks causing a bunch of ._foo files, but there's not too much you can do about that if you're copying data from HFS+ to something else. It's been a while since I used unison (I gave up my desktop, and now only use a laptop.), but seems like unison has the ability to actually merge files by fire off something like meld [sourceforge.net] to resolve conflicts. That's a big win.
Parent
Re:Unison UTF8 problem (Score:3, Informative)
That problem is mentioned all over the net, what's not mentioned is that the problem stems from transfers across different OSs.
A suggestion that might be a work-around : Sync Windows to a smb share, and use unison locally on that directory.
I am not using windows, so I'm fine. But, I also can't guarantee that my suggestion works.
dropbox? (Score:5, Informative)
www.dropbox.com
Re: (Score:2, Informative)
Re: (Score:3, Insightful)
Dropbox is good up to a few gigs, but their pay service is expensive per GB. I presently use it with SyncToy (scheduled) to keep stuff up-to-date between machines... but only for a small set of important stuff.
For larger things, the OP's selection of Jungledisk is far more cost effective... and if speed is an issue I wonder if he/she knows that Jungledisk backends to EITHER Rackspace or Amazon. They should try the other.
Re: (Score:3, Insightful)
Truecrypt is fine in these conditions. (Score:3, Interesting)
I have done what was described: rsyncing a TrueCrypt volume file that changes over time. Yes, there is an "expansion factor", but I would peg it at 2 - 5x the size of the diff within the encrypted volume. To restate: this scheme is entirely
wish USB was tougher (Score:2, Insightful)
USB sticks are a bit too fragile
damn straight. that is the number one problem with USB anything. i've seen more broken jump drives, and more broken usb ports from someone tripping over usb cable, than i care to fix. yes, they ARE handy as can be, but to WHOMEVER is designing USB 4 or whatever it will be called, PLEASE make the damn connection more sturdy.
Re: (Score:2)
With the price of USB sticks today you could just buy 10: use 1 to carry around in your pocket, put another one in an unused port on your main machine and put a script in a cronjob to synchronize regularly. If it breaks replace the main with the backup and plug a new one in the backup slot.
Re: (Score:3, Insightful)
WHOMEVER is designing USB 4 or whatever it will be called, PLEASE make the damn connection more sturdy.
They should make the connection more fragile, not more sturdy. The components should be sturdy, but if you design them to easily separate, things like tripping over the cable will cause little to no physical damage.
Apple's MagSafe power adapter is a good example of this. You can trip over the power as much as you want and it just disconnects. But neither the cord nor the laptop is damaged in any way and you're free to simply plug it in again and continue working. This should be the goal of every computer co
Re: (Score:3, Interesting)
All the little lock tabs on my RJ-45 ethernet jacks have broken off, so my network cables typically separate with minimum effort.
Magsafe had to be an Apple invention, though... since the older power plugs that came with them were way more fragile than any other laptop I've used. I've had to buy 2 replacement power supplies for my wife's old ibook already, and even now I still can't get power to it through the bent-up connector. It's a shame because it'd make a decent "Ken Burns effect" photo frame if I co
Dropbox (Score:5, Informative)
Chronosync, rsync, Jungle Disk (Score:5, Informative)
Jungle Disk is the best solution if you have more money than time and can rely on being online a reasonably good percentage of the time. In the Jungle Disk settings, you can specify a certain portion of your disk to be used as an offline cache. Jungle Disk will then keep downloaded files in that disk cache, so you don't have to worry about the download speed so much, assuming you have the disk space available for the cache. If you don't have the disk space you'll need an online solution anyway.
Chronosync is Mac-based, but you can set it up to sync your files with your Windows and Linux computers. I recently bought it and haven't been through all its features yet, but I'm pretty sure it will do everything you need it to for about $40.
Rsync is of course great if you have the time and expertise to set it up. If you want to take the time to learn how it works, it's probably the best solution. OTOH I think Chronosync will do 99% of what rsync will do (from what I've seen) and is easier so I felt like the purchase price of Chronosync was worth it to me.
SMB (Score:2)
Pick the laptop that you travel with (perhaps between home and work). Store all shared data on it. Samba mount that directory on the OS you are currently using (all of the OS's you list support samba).
You can work on your data on that laptop while on the move... but when you are using another computer (e.g. desktop) that laptop just effectively serves as an external disk. As long as you use GigE, there is no performance drop over a single locally-attached disk.
As an added benefit, you can backup this d
Another rsync like option (Score:3, Informative)
Unison from UPenn http://www.cis.upenn.edu/~bcpierce/unison/ [upenn.edu]
Works on all the platforms you mentioned... It can synchronize 2 disparate directory trees (you made updates to files A, B, C on one system and D, E, F on another system and want to merge them) and when it can't figure out what to do it asks you.
SVN+SSH (Score:3, Informative)
Set up one computer as a master repository and do everything over SVN+SSH.
Its what I do for the complete working set I have, passing between 3+ systems, is everything is through subversion over SSH to a backed-up system.
Distributed version control (Score:5, Informative)
Mercurial [selenic.com] for...
Windows [berkwood.com]
MacOS [berkwood.com]
For linux it should be in standard repos.
This way you can make changes on any of your systems and move them around as required, merging only when needed.
Re:Distributed version control (Score:4, Interesting)
I second this, although I use Git. Works great.
Parent
Proven technology (Score:5, Funny)
external media like USB sticks are a bit too fragile according to my taste
That's because you're using the wrong external media. You're going for new and exciting and not old and reliable. What you need to do is convert all of your data to human-readable binary (you know, 1100110001111 and so on), and chisel it on to large stone tablets. These are extremely resistant to wear, especially if stored with the written side down, and are virtually theft proof (who's going to steal a bunch of 500-pound rocks?).
You can also easily transfer data from one machine to another either via forklift or just by bringing the machines to the tablets themselves. Backup to Jungle Disk is simple: store a second set of tablets in the Amazon (you can even have a DR site in southeast Asia!), and hire local villagers to update them for you. You should provide a telegraph machine to the villagers to send data back and forth for synchronization purposes. If the telegraph is too unreliable, you can use an elaborate network of smoke signal or semaphore (the thing with the flags) specialists to send data back and forth.
Encryption may be a little more difficult, but easily solved if you happen to have a couple of old Enigma machines [wikipedia.org] sitting around (and really, who doesn't?). If the Enigma machine isn't reliable enough, Little Orphan Annie decoder pins can be used instead.
HTH.
Re: (Score:3, Funny)
SpiderOak (Score:3, Informative)
Dropbox (Score:3, Informative)
https://www.dropbox.com/ [dropbox.com]
"Free for Windows, Mac, and Linux"
Creates a folder that is kept in sync between different computers. You can share files with other dropbox users too via shared folders. 2GB or so of space is free. It also keeps multiple versions of files so you can go back to a previous version of a file if you need to.
I don't have any affiliation with the company, other than being a satisfied user.
Answers his own question. (Score:5, Informative)
I use a local svn repository for my work data. Now I need to frequently exchange and synchronize OpenOffice and Latex files and source code in various cross-platform programming languages between one machine and another.
Am I missing something here? What doesn't SVN do that you need? Clients exist for all named platforms and it handles OO, Latex, and source code files very well.
Traditional backup not always possible (Score:3, Interesting)
On a somewhat related note, my wife recently gave up on backing up to a different medium all together and now just builds backup into her work flow. She is a photographer and takes 50,000+ digital photos a year (in raw TIFF format). Each file is ~70MB so she requires several terabytes of storage. Loosing these images is unacceptable (ie: would result in a financial loss) and ripping to DVD for backup is simply impossible (would require a full time job in and of itself).
So she builds backup into her work flow such that at any one point in time she has at least one copy of the image spread across multiple drives. In the work flow, photos go from raw TIFF -> PSD pre-proccess -> PSD final -> JPEG for online viewing and she uses different drives for each stage in the process. That way, if a drive goes out, she only looses time, not data. For data retention requirements (~2-3 years), she just has ~10 TB of storage and rotates files from older jobs out onto other drives (spreading across 3 - drives as per the work flow).
Of course if something catastrophic were to happen to cause all of the drives to die (since they are all in the same location) then files would be lost, but that's what insurance is for.
NAS (Score:3, Interesting)
I looked into this problem myself, and it isn't pretty. I use a Windows box for everyday stuff, a Linux based server, and my Sister uses a Mac.
Basically I had the idea that I wanted to share some Anime videos with my Sister who also enjoys the TV shows. Anyway originally my idea was to buy an external USB HD, copy the files and then give her the HD, this way she also gets a new external HD out of the deal. However when I looked into it, not as simple as I thought. While most external drives are compatible with Mac or PC, the emphasis is on "OR". Usually they are formatted differently and are not compatible with the other once formatted.
The only other solutions I could find were to use a NAS (Networked Attached Storage), and this would work, because you would be using the network protocol to translate the data between OS, SMB I think it is called. Anyway at the time, they were very expensive and still are to a degree however they have come down in price more recently and I have seen solutions out there for 200-300$ range. Which is still more than I wanted to spend as an external HD only runs you about 100-200$.
I settled for the getto version and just spent the time and burned like 10 8GB DVD-R's as it was easiest. However for a continual solution, this would not work. I would say your only choice is some sort of NAS with SMB. I am sure given the time and know how you could also build your own linux box to do the same thing, but for price it likely depends if you have an extra kicking around someplace. Either could likely be set up for remote FTP. I would recommend just buying a NAS, likely simpler that way.
Two options for you (Score:3, Informative)
If you're willing to pay, you could use Dropbox. If you keep it under 2 GB, it's free. Anything you drag into your dropbox gets synced to their servers and then synced back down to your other PCs you have linked to your dropbox account.
If you would prefer a roll your own solution, and are willing to build a server, then go look at Novell's iFolder http://www.ifolder.com./ [www.ifolder.com]
Andy
Re: (Score:3, Funny)
RTFL. Jungle Disk has that covered. Throwing money included.
Rsync (Score:3, Insightful)
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
They are way beyond dicks and would go to unbelievable lengths not to provide you with the service you bought. They are proactively scanning your account and will shut you down (without ANY warning whatsoever) for any media file that look suspicious (that includes mp3's you recorded yourself, as in you're the artist).
At first when they started cracking down on people using disk space they just shut down everybody that was using "too much" in their opinion (they were specifically offering 1500 GBytes space a
Or dont' throw money (Score:5, Informative)
I;m a little confused if you are syncronizing, mirroring or backing up data. the title says synch, but time machine and jungle disk are more for backup, and at most can be used for mirroring a master rather than synching.
If you want to backup, take a look at Crashplan. It's got two unique features the other plans lack. 1) you can backup to your own physical media, not theirs. This solves the problem of how do you backup and restore say 200GB of data in less than a month to a remote service. if the disk is yours you do the initial backup locally, them move the disk to the remote location for incremental backups. And then you do the reverse when you need to do more than in incremental restore. 2) they will sell you just the software-- a one time cost-- and you don't have to pay for a monthly remote service cost unless you want to. in which case you just backup to a freinds computer that is hosting your disk. Your data is both safe and remote (and encypted) but you can also go get the drive is if you need it using your toyota rather than the internet.
On the other hand if you want syncronization then look into Unison. It appears the source forge project is not highly maintained but also mature enough to work well. it is cross platform and scriptable.
Parent
Re: (Score:3, Informative)
Seconded. I use Unison between Linux and OSX with no trouble, and used Win32 in the past as well. It isn't perfect, but it's closest by a wide margin.
Re:Or dont' throw money (Score:4, Informative)
Unison is quite stable and I use it daily to sync between Linux PCs over a network and Windows PCs via a USB key - add some flags to allow it make some decisions automatically and 99% of the time it works silently - add a cron/WindSchedule to it and it works a bit like Dropbox.
Parent