Forgot your password?
typodupeerror
Data Storage IT

Ask Slashdot: How Best To Synchronize Projects Between Shared Drive and PCs? 238

Posted by timothy
from the phrase-your-answer-in-the-form-of-a-cookie dept.
Koookiemonster writes "Our company has many projects, each one with a folder on a Samba drive (Z:\). Our problem is syncing only the programmers' current projects (~30 at any time) between Z:\ and their C:\Projects\-folder on five Windows 7 laptops. If we sync the whole Z:\-drive, our projects folders would be filled with too many subfolders, making it difficult to navigate. The folders contain OpenPCS projects (PLC) and related files (Word, Excel, PDF documents); a common project folder is 50 MB. Is there any easy to use, low-budget sync software with scripting, so that we could e.g. only sync folders that exist locally?" (Read more details, below, of what Koookiemonster is looking for.)
"Many programs do support selective sync, but choosing what to sync is awkward; projects and who works on them change daily. It is important that subscribing to a project is as easy as copying it from Z:\ to C:\projects\. The Z:\-folder with all of our current and past projects is located on a desktop PC running Ubuntu Linux. It can share files e.g. via Samba or FTP. All PCs are on the same (W)LAN. Off-site backups of Z:\ are taken care of via rsync. The company has three programmers, who usually handle their own projects alone, but very often others need to add files to projects. Bigger projects need more programmers. Currently we use FreeFileSync with a custom piece of Javascript to make batch files that synchronize e.g. folders C:\projects\123_ProjectName\ and Z:\123_ProjectName\ if the local folder exists. However, that solution lacks versioning, real-time sync and deletion support. It only syncs when we press a button, and then older files are overwritten by newer files (two way sync; older files go to a "sync-deletions"-folder).

PS. Bonus points for solutions that allow renaming project folders without renaming them on all laptops."
This discussion has been archived. No new comments can be posted.

Ask Slashdot: How Best To Synchronize Projects Between Shared Drive and PCs?

Comments Filter:
  • I don't know if this will do everything you need, but I have been using it at home to backup three machines to my NAS. Seems to work quite well.
  • by Anonymous Coward on Tuesday September 10, 2013 @01:34PM (#44809859)

    you are welcome

    • by camperdave (969942) on Tuesday September 10, 2013 @01:46PM (#44810023) Journal

      stop trying, use git instead

      ... or one of the many alternatives [wikipedia.org].

      • by jhol13 (1087781)

        I heavily recommend distributed version control system (git, mercurial, darcs, ...). You'll regret few years later if you don't.

    • Re: (Score:3, Insightful)

      by GameboyRMH (1153867)

      Correct answer almost on the first post, nice work ("svn" would also have been acceptable.)

      TFA is describing a horrible clusterfuck that must be replaced with proper revision control. You don't build on top of this, you KILL IT WITH FIRE!

      • by gnapster (1401889) on Tuesday September 10, 2013 @02:48PM (#44810783)
        svn is appealing in this case because you can check out subdirectories of a repo without downloading the whole thing, which is more similar to his current organization.
        • by achbed (97139)

          This. Git is great if you want to maintain a full set of everything on every system - every copy is a full backup. If you want to do selective work, then svn checkout of particular branches is the way to go.

          • by Gr8Apes (679165) on Tuesday September 10, 2013 @03:53PM (#44811765)

            You apparently don't have the slightest clue how git works, nor how to use it properly. I certainly am no git guru, but have no issues with using multiple git repos in projects, building sub-systems out of it, nor branching and merging, which it truly excels at. I wouldn't wish SVN on anyone that wants to use branching. It's about as brain-dead as it can get and still "function", barely. The only thing SVN has going for it is that it is almost atomic, whereas CVS, MKS, etc, absolutely aren't. ClearCase is a separate system that can be used, but requires a full time expert admin for anything more than basic code repos. At least it handles branching almost sensibly. Mercurial was in the running, but at the time git had (and still has) a larger user base with active improvements in various tools associated with it for my needs, thus I chose git.

            Learn a tool before you wrongly despise it publicly. You are wrong on every count:

            • You don't get a full copy of everything on every system if you don't want it. You can just check out a single branch.
            • Every copy is not a full backup. See previous line item.
            • Selective work is easier in git, you can clone the item(s) desired, branch locally, and merge when complete, pushing only if desired.

            Lastly, you can essentially "checkin" every change for a full history of what you did on a local branch, revert, merge other branches, etc, with no effect on the main dev branch(es). And you can do all this without even being connected to a "main" server. Maintaining parallel branches with constant merges is cake compared to SVN and other central repo schemes. There really is no situation where I'd rather use SVN or anything like it that I can think of, when I have a choice.

            • " There really is no situation where I'd rather use SVN or anything like it that I can think of, when I have a choice."

              So can git already manage binary blobs in any sensible manner?

    • by Darinbob (1142669)

      Or just about any source code control system (most allow binaries as well). I can't think of any that don't allow syncing just files that are wanted. Further, these systems are designed to safely share the files, not just "sync"; files can be locked and made exclusive, or conflicts between multiple authors resolved, etc. This is essentially technology that's 40 years old.

    • I wouldn't use git. Git requires you to always clone the entire repository. That's fine if the repository is just source code and text files, but the more binary files you have the less attractive this can be.

      I still favor svn over git myself, but it just suits my workflow better than git.

    • git is a rather poor choice for binaries due to the way it stores things. SVN is the better choice for binaries, especially for a situation where you want a central repository server.

      But really, the answer is "nearly any centralized version control system" is a better choice then trying to rsync/unison stuff around. With the advantage that everyone *knows* they have the latest (by asking the server) and you get full revision history for everything.
    • git, rsync, etc. are all excellent options.

      Why on earth is anyone sync'ing server folders in 2013 anymore?

  • by charnov (183495) on Tuesday September 10, 2013 @01:36PM (#44809877) Homepage Journal

    https://owncloud.org/ [owncloud.org]

    Keep the masters in a private cloud and sync to it from your PCs. Git and other multi-user SVN is an idea, too. Also, SharePoint is excellent (but lots of overhead).

    • by Amouth (879122)

      I'd like to chime in on ownCloud. It works but it also doesn't

      Owncloud fails in a large multi user environment over low speed connections, and with office files if the users are working in the same folder that syncs. We tried it, and for single in a folder it worked great, once we added others (especially with low speed or low quality connections, aka traveling users) it failed miserably. We had many many many conflicts for no reason. many many 0 byte files do to cross sync issues.

      We ended up canning it

      • ive been playing with owncloud for the last week, it's awful, it loves just deleting all my files or corrupting them :(

      • by nschubach (922175)

        I've found Bittorrent Sync great for selective syncing... You select a folder to sync and it gives you a hash that you can give to someone else who just needs that hash and they can specify any folder for those files. If you have a project wiki or something you can just put the hash alongside the project name and anyone that wants that synchronized folder can put in that hash. It does LAN syncing so it doesn't always have to hit the web, you could setup a server assign each project to a sync target and sh

        • by Amouth (879122)

          I got in on the Alpha of that, but didn't have time to fully review it. I will have to go back and look at it.

          Thanks,

    • We ditched OwnCloud for GoodSync. OwnCloud in theory works OK. However, if someone with a laptop is away from the Internet, it is real bad about losing all the client settings and just quits syncing. Also, in the right circumstances, it can generate a huge amount of files by versioning during collisions.

      GoodSync is not perfect, either, but it is far superior to OwnCloud. Is it free? No. Work better? Yes.

      In theory, GoodSync has support. I was not thrilled with it. We kind of got cross and they thought I was

    • by siDDis (961791)

      Stay away from Owncloud....really, those guys don't even know that all modern filesystems have an API for watching changes in your filesystem. And syncing works on file level instead of block level.

  • by Anonymous Coward on Tuesday September 10, 2013 @01:37PM (#44809889)

    This sounds like the ramblings of someone who has never heard of revision control.

  • by the eric conspiracy (20178) on Tuesday September 10, 2013 @01:39PM (#44809923)

    USE VERSION CONTROL!!!!

    Git
    Mercurial
    Perforce
    Subversion
    Vesta
    CVS
    ClearCase
    VSS
    StarTeam

    The choices are legion. What you are doing is not a choice.

    Pick a version control system and your life will be much easier (after the learning curve).

    • by Dieppe (668614)

      USE VERSION CONTROL!!!!

      Git Mercurial Perforce Subversion Vesta CVS ClearCase VSS StarTeam

      The choices are legion. What you are doing is not a choice.

      Pick a version control system and your life will be much easier (after the learning curve).

      I think this story can be closed with this post right here. Version Control, end of story. P

      • Re: (Score:3, Funny)

        by Anonymous Coward

        Why close? We can now start the entertaining flamewars about which version control software is the only reasonable one! ;-)

    • by the eric conspiracy (20178) on Tuesday September 10, 2013 @03:29PM (#44811379)

      I've used many of the above. Here are some comments (the first four are free):

      1. Git - really capable tool, with opaque command model. Maybe not for newbs. Managers hate using it (one reason to like it). Has good cloud support with GitHub.

      2. Mercurial - pretty easy to use. Great if your projects don't have large binary files. It's a good introductory decentralized repo tool.

      3. Subversion - pretty good if you want a centralized repo.

      4. CVS - your father's version control system. Still way better than not having version control.

      5. ClearCase - good in the hands of a ClearCase specialist. Would not be my choice for a small team of version control newbs.

      6. VSS - My experiences with it were not good. I hear it's improved. I'd look at it ONLY if I was in a Microsoft only shop.

      • by Richy_T (111409)

        It's time someone constructed a version-control file system. I mainly use subversion and it can be a real pain when you used mv instead of svn mv. (And other related commands). These things should be transparent (not that that's an indictment of Subversion).

        • by Richy_T (111409)

          Keeping two copies of everything is also needless.

        • Maybe btrfs... can't remember if that is in the spec.

          You can also do fun things with the various VCS such as mounting them as file systems (svnfs, gitfs, etc.).
        • by jrumney (197329)

          It's time someone constructed a version-control file system.

          That's basically what Clearcase is. And why it needs a Clearcase specialist to administer it.

      • 6. VSS - My experiences with it were not good. I hear it's improved. I'd look at it ONLY if I was in a Microsoft only shop.

        VSS is a deprecated tool at this point, last updated in 2005, and only having extended support from Microsoft at this time. Team Foundation Server [wikipedia.org] is Microsoft's current source control offering.

  • Not sure if he is trolling, or if he has deeper issues that can't be solved with version control systems like git or hg (or even good 'ol CVS!).
    • by Darinbob (1142669)

      I wonder how a company even gets into this situation. Granted, I don't think many schools teach the fundamentals of how to actually manage a project but I think even in the greenest of startups there should be at least ONE person who's held a real job before and has seen this. Nothing here is new. SCCS was started in 1972. Having multiple programmers on a project has existed since computers were invented.

      My guess is that you get some clueless business oriented guy making a startup ("I'll get rich, my id

  • by Empiric (675968) on Tuesday September 10, 2013 @01:43PM (#44809981) Homepage

    Sounds exactly like what an RCS like Subversion is good for.

    Each user pulls down the directories relevant to him/her from the overall repository, updates at will from the central source, and pushes up changes at will with a couple of mouse clicks.

    TortoiseSVN will even give you handy little icons on your local folders in Explorer to tell you if what you have in your local directory isn't synced with the central server, and it's two clicks to make that happen. I actually think you don't want to "force synchronization" on an ongoing basis, seems like a great way to overwrite a lot of your developers' (and others') ongoing work.

    • by scrib (1277042)

      Yours is the answer I wanted to write and is the solution I've used for just such situations. However, it violates one of the first ridiculous requirements: "It is important that subscribing to a project is as easy as copying it from Z:\ to C:\projects\."

      Koookie, that may be an easy way to copy files, but it is a miserably difficult way to manage projects - even for developers. SVN takes a little time to set up and understand and requires maybe a day's worth of training for any developer who hasn't used sou

      • by djmurdoch (306849)

        If the admin keeps Subversion working copies of each project in Z:\, then developers can indeed just copy the top level directory of each project from Z to C and get their own working copy.

      • Our approach (we use SVN) is to create working copies for every single client under a C:\Clients\xyz scheme. Where 'xyz' is the 3-letter codename for each client.

        Each of those working copies is created with a script (stored in a 'svn-scripts' repository) that does a "file children only" checkout. The svn command for that is "svn checkout --depth files URL targetdir".

        Because we only bring down immediate file children in the root of the repository, it doesn't take up much space. And bringing down a par
  • by gmezero (4448) on Tuesday September 10, 2013 @01:43PM (#44809987) Homepage

    We used to use ViceVersa Pro to sync our team but eventually moved over to Plastic SCM which has been friggin' awesome. It not only supports code, but also art assets. Plus it has the best support for branching. One team can be working on a branch specific to one project, while another works on a second branch while the main trunk stays clean and build-able. You can even have developers run their own local repository on their desktop/laptop and have them replicate/merge either on a schedule or when they connect to the LAN (if they work offsite alot).

    You only bring stuff back to the main trunk when you're ready to merge a branch back in. You can even merge branches separate from the trunk. We check everything into it, code, art, and our Doxygen output. It's been a time save on orders of magnitude.

    Not trying to be a sales pitch, but you should check it out.

  • by MobyDisk (75490) on Tuesday September 10, 2013 @01:50PM (#44810079) Homepage

    Syncing files like this is a mess. Perhaps you should look beyond share drives. You are trying to solve the technical problem, but if you step back you might see a business problem. Consider 3 alternative approaches:
    1) Keep the files on the share drive and do not mirror them locally.
    2) Use a source control system (Ex: GIT, TFS, Subversion)
    3) Use a groupware / content management system / document management system. ( Ex: Sharepoint, Confluence, QDMS, Lotus Notes, Microsoft Exchange, Drupal, SAP, Groupwise)

    Knowing the right terms helps find the software you need. Here are some links to Wikipedia which has the right terms, and some lists of software:
    http://en.wikipedia.org/wiki/Content_Management_Systems [wikipedia.org]
    http://en.wikipedia.org/wiki/Document_management [wikipedia.org]
    http://en.wikipedia.org/wiki/List_of_collaborative_software#Comparison_of_notable_software [wikipedia.org]

  • You have programmers. You have multiple projects. They might be working offline. For this, you really need a Distributed Source Control system such as git or mercurial. I personally recommend mercurial as it's got good Windows tools (TortoiseHg and HgScc for Visual Studio integration). You can put your "pure" repository on your share, then have the programmers push to it -- or, better yet, have an "incoming" for each project to which anyone can push, then a "pure" to which only project leads have write acce

  • If you hire experienced, competent programmers, they will be able to solve this issue for you. First they will suggest using version control (it seems frighteningly likely from your writeup that you're not currently using it). Probably git, but there are other good ones.

    At that point the problem will become redefined. What you want is a script that:
    - Iterates through the local working directory.
    - Finds project folders that are NOT being worked on, and are also currently clean (no uncommitted files),
  • It's sad (Score:5, Insightful)

    by msobkow (48369) on Tuesday September 10, 2013 @02:03PM (#44810223) Homepage Journal

    It's sad to see an entire team of supposedly "professional" developers which have never heard of version control.

    You can't even blame it on the Windows environment -- MSVS supports hooks for several version control systems either natively or through plugins/addons.

    This whole story just reeks of some manager saying "We can't afford to set that up -- it would take too much time" any time someone has suggested it.

    Because I flat out refuse to believe the entire team doesn't know any better.

    • As soon as I saw all those drive letters in the guy's summary I just knew where this was going... very sad indeed when coders don't have a clue about the tools that obviously suit them best.

    • by jon3k (691256)
      Me thinks no one asked the programmers. Some IT helpdesk guy is trying to solve it.
    • I do if they're microsofties. Total idiots - the lot of 'em.

    • by Darinbob (1142669)

      I was at one company where most programmers were independent. It was sort of an odd business model that derived from the mainframe world; an author would write an application, write the docs, package it up, maintain it, and also collect a declining percentage of royalties. If the program was successful you could hire a developer to assist.

      So my boss and I were the only people working on 2 programs. My boss learned with a "Learn C in 20 Days" book and previously had only done mainframe assembler and servi

    • Re:It's sad (Score:5, Interesting)

      by WuphonsReach (684551) on Tuesday September 10, 2013 @06:15PM (#44813695)
      Because I flat out refuse to believe the entire team doesn't know any better.

      It's way more common then you'd think. I worked for a company back in 2000, with close to 20 developers. Not a single one of them knew how to do version control. Instead, they all resorted to making copies of files, tacking on dates or times or just numbers, and relying on the backup tapes if they had to undo a particular change. It was hellish and you'd hear at least one conversation per week where they were trying to figure out who had the latest version of XYZ.

      I rolled out VSS (hey, it was 2000, our choices were limited) with SourceOffSite for the remote workers. If I did it today, it would be either Hg, git or svn.

      Very few schools back in the 90s or early part of the decade taught VCS concepts or forced students to use them. This is slowly changing with the advent of things like 'github' which has a big mindshare and introduces people to the concept of VCS.
      • Yep. In a former job at a very large financial services firm, I worked on their source control/build/packaging/deployment systems, and I was stunned by the number of developers who not only resisted using source control, but actually would not even acknowledge the value of it. One time, I raised gasps in a meeting of several dozen devs by making the statement "If you have a problem using source control, you have no business being a professional software developer." Yes, that was a controversial statement

  • by zzsmirkzz (974536) on Tuesday September 10, 2013 @02:06PM (#44810269)
    If you don't want to do Version Control as others have suggested then I recommend Super Flexible File Synchronizer. It is a great product with lots of options in regards to what does and does not get sync'd. It is inexpensive to boot. http://www.superflexible.com/ftp.htm [superflexible.com]
  • I suggest finding the person who says you can't use version control, and locking him in a room until he either changes his mind or ceases to be a problem. Then, use version control.

  • http://www.rsnapshot.org/howto/ [rsnapshot.org]

    Well, actually probably rsync will be sufficient for your needs. And rsnapshot is probably a little more than your needs. I suppose that only thing you need to configure is a rsync server on windows, a nice writeup you will find here: http://www.stillnetstudios.com/snapshot-backups-howto/ [stillnetstudios.com]

    I recently used this to configure my wife's windows PC, so that it will work with rsnapshot, and backup all her projects. After configuring rsync server on windows, the rsync operation
  • You said you have programmers working on these projects. They probably each have their own preferred way to do this, why not ask them? If they can't come to a consensus, you could have them write their own solution.
    • by Bengie (1121981)

      If they can't come to a consensus, you could have them write their own solution.

      If they couldn't do the first part, then they're probably not smart enough to do the alternative.

  • An analogy. (Score:5, Funny)

    by pclminion (145572) on Tuesday September 10, 2013 @02:49PM (#44810803)
    "I'm a contractor. I have a team of carpenters who are tasked with building a house. It seems this is going to require the driving of a large number of nails. My team of carpenters would like to know what sort of tool or mechanism would work best to drive these nails. Right now, we have one guy who holds the nail while another guy hits it with his thermos. This does eventually drive in the nail, but 90% of the time the nail bends, and it's denting our thermoses. I wonder if there exists some genius, super-carpenter bad-ass out there who might be able to suggest a better way."
    • by vux984 (928602)

      I'd been drinking coffee, I'd be wearing coffee.
      Thanks for this absolutely perfect post.

  • I second git. git scales badly with large files. If this is a problem, you could have a look at git-annex http://git-annex.branchable.com/ [branchable.com] The concept needs some time to grasp, but it's really powerfull.
  • by Kaz Kylheku (1484) on Tuesday September 10, 2013 @02:56PM (#44810877) Homepage

    Git is absolutely not a good first version control system for people who are clueless about version control. (Such as, evidently, your developers).

    Git requires prior experience with at least two simpler version control systems. In git, you often run into scenarios that require you to understand its complicated repository representation so that you can choose the best steps to unravel them, based on understanding the ramifications of each approach.

    The implementation of git is not hidden from the user behind a robust set of "no brainer" use cases.

    The decentralized model alone will confuse the heck out of workers with no prior version control experience.

    Use a system that has a centralized server from which working copies are checked out, like Subversion.

  • SVN or Git for code repository. Easy to set up, lots of windows and linux tools, command line based even if you want that.

    However for documentation, i recommend confluence, or the many free wiki based collaborative solutions. This allows people to post on a wikipedia like site their documents. They become automatically searchable, people can collaborate in the documentation with version control built in. Confluence allows you to drag and drop ppt, import and export word documents, drop excel files in th

  • The obvious "Use Version control/Git" is actually and improvement of your methods.
    Here is an actual to the point answer to your question about software. A requirement of real time-sync was mention.

    There is Synkron [sourceforge.net] (has a scheduler) , OneSync [google.com] (has real time sync), and Unison [upenn.edu] (uses rsync).

    All are open source and free. Have fun.

  • Sparkleshare (http://sparkleshare.org/) is a "transparent" front end for Git which turns it into a simple file sharing tool. This would probably be appropriate for most of the actual "file sharing" applications the OP mentions (gaining many of the advantages of Git while keeping the complexity hidden until its needed), while obviously any source code fprojects should find their way into some kind of version control repository, probably Git as well, with TortoiseGit (http://code.google.com/p/tortoisegit/) be

  • EMC Syncplicity [syncplicity.com]. Note: I am the desktop client lead.
  • You might want to checkout git-annex: http://git-annex.branchable.com/ [branchable.com]

    It handles the idea of larger repositories with disconnected parts. You get git versioning of files and the ability to replicate portions of the data at will.

  • The developers at my company, who use windows laptops, keep a lot of their code in VisualSVN server [visualsvn.com] and then use eitherAnkhsvn [collab.net] to do the checkout straight into Visual Studio, or TortoiseSVN [tortoisesvn.net] if you want to be able to right-click in any folder and checkout to that location.
  • by hobarrera (2008506) on Tuesday September 10, 2013 @06:49PM (#44814041) Homepage

    I agree that a VCS is the best choice, as suggested above. If, for some reason, you don't want to use that (eg: too many binary files grow you repo too much), unison [upenn.edu] is a great choice. It uses rsync to sync files, and keeps track of which files where modified on which side.

  • I'm curious about a couple of things. Of course pretty much everyone here is screaming "source control" But how is that you have a programmer working on THIRTY projects at one time? Perhaps I'm misunderstanding your use of "project" but I think I would go crazy if I was trying to juggle thirty different projects. Perhaps some sort of consolidation is in order.
    You also mention if you sync the whole thing, navigation would be a problem with too many subfolders. A good source control will help alleviate this

Remember: use logout to logout.

Working...