How Do You Manage Dev/Test/Production Environments? 244
An anonymous reader writes "I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs through the different environments. I'm finding as our company grows (we currently host 50+ sites) it is cumbersome to manage all sites by hacking away at the command prompt. I would like to find a solution with a relatively easy-to-use user interface that provisions dev/test/live environments. The Aegir project is a close fit, but is only for Drupal sites and still under heavy development. Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
Separate SVN deploys (Score:4, Informative)
happy with phing (Score:3, Informative)
There's really only so much you can do generically. I'm really happy with phing. I use the dbdeploy task to keep my databases in a similar state. I build on a local machine, deploy via ssh and then migrate the database.
I'd suggest that rather than checkout at each level you create a continous integration machine using something like cruise control or bamboo, then push out build tarballs and migrate the database.
Re:Have the hosts email problems to an email accou (Score:4, Informative)
Never heard of a loghost eh?
SVN etc. (Score:3, Informative)
Still not figured out an efficient way to version MSSQL and MySQL databases using OSS, though. Open to suggestions!
Puppet (Score:2, Informative)
Tools, Practices and Standards (Score:5, Informative)
We utilize a number of tools depending on the site, but generally:
Version Control (Subversion) for management of the code base (PHP, CSS, HTML, Ruby, PERL,...) - http://subversion.tigris.org/ [tigris.org]
BCFG2 for management of the system(s) patches and configurations (Uses svn for managing the files) - http://trac.mcs.anl.gov/projects/bcfg2 [anl.gov]
Capistrano/Webistrano for deployment (Webistrano is a nice GUI to capistrano - http://www.capify.org/ [capify.org] / http://labs.peritor.com/webistrano [peritor.com]
However, all of the tools above mean nothing without defining very good standards and practices for your organization. Only you and your organization can figure those out...
Check out Springloops (Score:4, Informative)
It's hosted Subversion, with a slick web interface that walks you through darn near everything. You can configure development / test / production servers that can be accessed via FTP or SFTP and deploy new builds to any of them with just a couple of clicks. It integrates with Basecamp for project management, and it is really cheap - it sounds like either their Garden or Field plans would meet your needs, and they're both under $50/month.
Check them out here. [springloops.com]
Not affiliated with them in any way, other than as a satisfied customer.
Look at Capistrano, steal ideas from Rails (Score:5, Informative)
Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.
Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.
You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.
I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.
Packaging Packaging Packaging... (Score:5, Informative)
Its amazing, how this seemingly obvious question, always gets weird and overly complex answers.
Think about how every unix os handles this. Packaging!
Without getting into a flame war about the merits of any packaging systems:
- Use your native distributions packaging system.
- Create a naming convention for pkgs ( ie, web-fronted-php-1.2.4, web-prod-configs-1.27 )
- Use meta-packages ( packages, whose only purpose is to list out what makes out a complete systems )
- Make the developers package their software, or write scripts for them to do so easily ( this is a lot easier than it seems )
- Put your packages in different repositories ( dev for dev servers, stg for staging systems,qa for qa systems , prod for production, etc et c
- Use other system management tools to deploy said packages ( either your native package manager, or puppet, cfgengine, func, sshcmd scripts, etc )
And the pluses? you always know absolutely whats running on your system. You can always reproduce and clone a systems.
It takes discipline, but this is how its done in large environments.
-
Re:Separate SVN deploys (Score:3, Informative)
Do _not_ use Subversion for this. Use git, even if you have to use gitsvn to point it to an upstream Subversion repository. Subversion's security models in UNIX and Linux are exceptionally poor, and typically wind up storing passwords in clear-text without properly notifying you. (Now it notifies you before storing it, but usex it automatically.) Subversion also has very poor handling of multiple upstream repositories, and there is no way to store local changes locally, for testing or branching purposes, and only submit them to the central repository when your changes are complete.
git is faster, lighterweight, and performs far better for distributed systems each of which may require local configurations.
Quick Brief (Score:5, Informative)
Develop 4 Environment Structures
Development (DEV)
Integration Testing (INTEG)
Acceptance (ACPT)
Production (PROD)
For each system create a migration script that generically does the following:
(We will use SOURCE and DEST for environments. You migrate from DEV->INTEG->ACPT->PROD)
The migration script as it's core does the following:
1) STOP Existing Services and Databases (SOURCE and DEST)
2) BUILD your deployment package from SOURCE (This means finalizing commits to an SVN, Creating a dump of SOURCE databases etc.) If this is a long process then you can leave the DEST running and STOP DEST at the end of the build phase. I do this as builds for my world can take 2-3 days.
3) CONDITION your deployment package to be configured for DEST environment (simple find and replace scripts to correct database names, IP address, etc. These should be config files that are read and processes.) This is common if there are different security SAPs, Certificates, Etc that need to be configured. For instance you may not have SSL enabled in DEV but you might in INTEG or ACPT.
4) BACKUP DEST information as an install package(this is identical to the BUILD done on the source. This BACKUP can be deployed to restore the previous version.) This should be the same function you ran on SOURCE with a different destination (say Backups verus Deploys)
5) MIGRATE the install package from SOURCE to DEST
START DEST
6) TEST TEST and RETEST
7) If all tests pass then APPROVE. This is the green light to re-start the SOURCE services so development can move on.
That is a brief of my suggestion.
DEV is obvious ... yeah...
INTEG is where you look for defects and resolve defects. Primary testing.
ACPT is where user and BL acceptance testing occurs and should mirror PROD in services available.
PROD
I handle about 790+ applications across 2000+ pieces of hardware so this may appear to be overkill for some but it can be as simple as 4 running instances on a single box with a /DEV/ /IT/ /ACPT/ /PROD/
Directory structure with MYSQL running 4 different databases. The "Script" could be as simple as dropping the DEST database and copying the SOURCE database with a new name. Other options are creating modification SQLS for instance that are applied onto the exist database.
e.g. STOP, UPDATE, START
to preserve existing data. In the case of Drupal your DEV might pull a nightly build and kick out a weekly IT, a biweekly ACPT, and a monthly PROD update.
JUST REMEMBER THAT YOU MUST MAKE SURE THE PROCESS IS ALWAYS REVERSABLE!!
The script to deploy needs to handle failure. There has to be a good backout.
You should have a method to backup and restore the current state. Integrate that into the script. Always backup Before you do changes and AGAIN after you change. DEV may need to look at the failed deploy data (perhaps a substitution or patch failed, they need to find out why.)
Before Backup and After Backup in the migration script.
And always 'shake out' a deployment in each environment level to make sure problems to propogate. You find problems in IT, you test to make sure what you found in IT is resolved in ACPT. Your testers should NOT normally be finding and filing new defects in ACPT environments with the exception of inter-application communication that might not be available in earlier environments. (Great example might be ACPT has the ability to connect to say a marketing companies databases where you use dummy databases in IT and DEV.) 80/20 is the norm for IT/ACPT that I see.
Good luck. Use scripts that are consistent and invest in a good migration method. It works great for mainframes and works great in the distributed world too.
A special condition is needed for final production as you may need temporary redirects to be applied for online services (commonly called Gone Fishing pages or Under Construction Redirects)
Re:Packaging Packaging Packaging... (Score:3, Informative)
Re:Separate SVN deploys (Score:5, Informative)
Auth token caching can be easily disabled and svn export, not svn checkout, should be used for deploying test/prod environments (like I've seen way too many people do).
Git (or any other distributed version control system) is great if you are into distributed development, but don't blame the tool when you don't know how to use it properly or expect it to be something that it's not.
Re:Separate SVN deploys (Score:3, Informative)
What do you mean, git doesn't integrate with Apache? It works well as an Apache client, there's 'viewgit' if you need a bare web GUI. And for this purpose, locally recordable changes seems critical.
Re:Separate SVN deploys (Score:3, Informative)
http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt [kernel.org]
trac does have git support too, etcetera
From professional experience: (Score:3, Informative)
I have adapted my system from the 5 years I professionally did it.
First of all, it's a 3-stage system.
You have a couple of live servers, a identical staging server, and the user machines.
Every system has a clone of the files. the servers have rsync copies of the stage server files.
And the users all sync to the stage with GIT.
Everyone has a local clone of the stage server software too, so he can test server-side code right on his machine.
That's important in every company where people could do conflicting (and even big, global) patches.
The stage server then has validity tests running. Compilations and unit test cases wherever possible. Including the database, the server side code, and rendering test pages in all relevant browsers, to diff the rendered versions (images) of the pages. (There's a app for that in Firefox, but otherwise it's desktop automation.)
There's an red alert box in the test case overview when something fails. Which gets checked every evening, before pushing anything to the live servers at night.
The only thing that turns out to be a bit hard, is to test the client-side logic (e.g. of web-apps) in a transparent manner (= keeping the software configurations and serveride code the same to be able to rely on it).
Then there is a emergency push and and emergency direct live update mechanism, for cases when you quickly have to fix something that got overlooked. (Which usually should result in a new test case to be written, to catch all such problems.)
A well-integrated project management system is very important. At the end of my first company, it was a self-written one with good integration. But in the beginning, something like Trac might suffice.
Then very important is, to have a knowledge base for all things that need to be remembered. Like a meta-documentation. Workflows and procedures. Why the mysql server will not restart on a reboot of stage server clones. Little hooks and mantraps like that. I recommend a Wiki.
And last but not least, never ever forget to have a Bugzilla. If you're good, you can integrate Bugzilla, the test validations and the task/project management into one system. Making the validity tests create bugs in Bugzilla, and bugs being the same as tasks (which makes test-driven development easier).
Yet this all is completely worthless, if your colleagues don't use it! ;) ...". It's the mother of all PHB stories. ^^
Unfortunately, I learned, that when someone can/em> do something wrong, he will.
So if you can't lock down possibilities to only those required, you have to be very very careful with who you hire. Especially with "web development", where you get sinology students who learned HTML while working as a taxi driver, stating that they are "professional web developers with 5 years of experience", while honestly believing that. And team leaders believing it too, because they are just as "competent". Because they themselves either started as something an simple as link collectors, or the boss of the company does not know shit about his business, and hired those types. They then usually get promoted to "Head of
The key is: Make them like to work the proper way. If nothing helps, money can always push them in the right direction. It's called "bonus". :)
And making it their project too, by also embracing their decisions!
Re:SVN etc. (Score:3, Informative)
Deployment to live servers via SVN checkout when the time comes
Side note: I humbly suggest (as someone else mentioned elsewhere) you use export instead of checkout for the live deployments.
Re:Mod Parent Up! (Score:2, Informative)
Basically, the writer said he wanted to "give back" by using the software.
That is an interesting take on what what "I would much rather use something OSS so I can give back to the community" means. Sure, it could mean what you said, but I suspect it was meant more in the spirit of having the opportunity to report any issues with the program back to the author(s). Perhaps I am reading into his statement as well...I don't know. What I do know is that he clearly hit one of your hot buttons.
Re:Separate SVN deploys (Score:4, Informative)
svn+ssh doesn't store anything in clear text. If that's a security concern for you, there's already a solution in place. Git is not the be-all and end-all solution to source control; it does many things very well, but there are a few things it does very poorly (repository control; with git, developers have a local copy of the repository which means that a stolen laptop comes with complete revision history). When the systems you're working on have certain sensitivities (legal, patent, security, etc), this can be a major weakness.
We do something very similar to the original submitter at work. We have 10-15 project branches open at a time. For us, we make sure that our code is subdirectory-agnostic (meaning it can run on the root of the website, or it can run out of a subdirectory). We use directory paths for branches, and we use internal DNS records for environments. http://de-appname/branchname [de-appname] would be a development branch while http://va-appname/branchname [va-appname] would be a validation branch.
For our lifecycle, we have development, validation, staging, pre production, and production. Development and validation are the only branching locations; staging, pre, and production are each single-path locations (though staging is a branch reserved for this purpose, pre production and production are /trunk)
On http://de-appname/ [de-appname] and http://va-appname/ [va-appname] there is essentially a directory listing along with the fully qualified branch name, revision, most recent contributor's name (even spelled out by looking up their record in LDAP), commit time, and most recent log message. Developers get a drop-down menu next to project branches which they can use to update the working copy there on that shared server (does a little ajax call and shows you the result in real time as though you were at a terminal). You can also create clean checkouts and even create new branches (either off of trunk or off of another branch). Finally you can even close a branch through this interface; it deletes the branch with a meaningful log message, and cleans up the files on that server. All through a web interface, no need to remotely log into a machine for this purpose. There's no reason for someone to administer this, each developer creates a working copy when and where he thinks it makes sense for himself.
Because our back end is SAP, we don't have to deal with multiple database environments. There's no "create a new copy of SAP" - indeed when this is something that's organizationally important (testing a major upgrade), it's a multiple day long process. We have a fixed set of data environments, and these are tied to hostname (going back to de-appname, va-appname, stage-appname, etc). If there were multiple database environments to worry about (eg, you wanted to be able to effectively branch database environments too), it wouldn't be a huge deal to set up a template database and have the same scripts we use to manage branches through a web API clone that database and update config in the appropriate app.
The key that I'm trying to get at is that you should create a web based tool to allow developers to manage this themselves. The developers will thank you because they'll be able to get what they want faster than you could have provided it, and they'll have control over when where and why working copies of their work in progress appear.
Re:Separate SVN deploys (Score:3, Informative)
There are so many things to do to get this right - basically I'd add that you need a three-silo people model to match the three silos of dev,test,prod. Your development side are the creative ones; give them the tools they ask for and let them play. You need a critical, intelligent and demanding test manager in the middle, and for the production gatekeeper you need someone with absolutely no imagination at all (follow the rules, tick the boxes, or *zero* chance of advance to production. Seriously. Tell the people who think otherwise the *boat*will*sink* if all the test boxes aren't ticked. Just don't try to run development that way, mutiny isn't pretty.)
For the test through production phases, make sure you have your servers virtualised. Test because test environments tend to proliferate (try, just try to keep a baseline somewhere? Careful when you update it...) and Production because you'll need speedy rollback if you auger it, and strict version control on the server versions. This applies whether your virtual:physical server ratio is 1:1 or otherwise.
Look into data de-duplication solutions to keep the total disk space used by virtual images down.
ITIL is nice but it's really only the scaffolding. You still have to provide the cathedral.
Re:Separate SVN deploys (Score:3, Informative)
You wrote:
> You can say exactly the same about any form of password caching. You don't use Firefox because it can store your passwords?
No, I can't say exactly the same thing. Subversion, by default on UNIX and Linux, stores the passwords in cleartext. That is one of the stupidest things I've ever seen for allegedly "enterprise" class software. The only other source control I've seen do something that stupid is CVS, which is what Subversion is descended from.
"Configure in your dev environement" means that your dev environment has the configurations checked out, as a Subversion environment, not an export. This means that your dev environment has precisely the problems I mentioned, and your production environment cannot make _any_ recorded changes, not even local configuration changes, without routing them through the development environment. This is, in practice, insane, because it inserts a full testing and commit cycle to what should be a very modest in-place repair or a modification that belongs only in the production branches, not the development branches or environment.
Yes, I've had to deal with this. It is fundamentally an error to rely on pure 'svn export' for configuration management.