6515739
story
Comments: 244 +- Ask Slashdot: How Do You Manage Dev/Test/Production Environments? on Tuesday October 20, @12:58PM
background: url(//a.fsdn.com/sd/topics/topicit.gif); width:75px; height:61px;
it
An anonymous reader writes "I am a n00b system administrator for a small web development company that builds and hosts OSS CMSes on a few LAMP servers (mostly Drupal). I've written a few scripts that check out dev/test/production environments from our repository, so web developers can access the site they're working on from a URL (ex: site1.developer.example.com). Developers also get FTP access and MySQL access (through phpMyAdmin). Additional scripts check in files to the repository and move files/DBs through the different environments. I'm finding as our company grows (we currently host 50+ sites) it is cumbersome to manage all sites by hacking away at the command prompt. I would like to find a solution with a relatively easy-to-use user interface that provisions dev/test/live environments. The Aegir project is a close fit, but is only for Drupal sites and still under heavy development. Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
Related Stories
Submission: How do you manage dev/test/production environments by Anonymous Coward
When a girl can read the handwriting on the wall, she may be in the wrong
rest room.
All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.
How slashdot does it (Score:5, Funny)
How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
I do the same as Slashdot.org does - Make the changes on live code, except a little downtime and weird effects and then try to fix
it - while actually never fixing it. After all the results are not that significant:
- if someone posts about it on a thread, mods will -1 offtopic it and no one will hear your complain
- many people will "lol fail" at the weird effects, like when kdawson decides to merge two different stories together [slashdot.org]
Re:How slashdot does it (Score:4, Insightful)
That's not that far from the truth in MANY places and projects I've seen.
I've actually come to the conclusion, that on many govt/DoD projects, that the dev. environment in fact becomes the test and production environment!!
I learned that it really pays, when spec'ing out the hardware and software that you need, to get as much as they will pay for for the 'dev' machines....because, it will inevitably become the production server as soon as stuff is working on it, the deadline hits, and there is suddenly no more funding for a proper test/prod environment.
Parent
Slashdot and this company ... (Score:3, Funny)
You are not a n00b (Score:5, Insightful)
You may be a new system administrator, but you are not a n00b.
A n00b wouldn't realize he was a n00b.
Re:You are not a n00b (Score:4, Interesting)
If he is indeed allowing FTP logins over the public Internet (as the submission suggests), he is a n00b whether or not he realizes it.
Parent
Re:You are not a n00b (Score:5, Insightful)
Or maybe they're not over public internet, or are tunneled, or they're sftp and he just calls it ftp? We dunno.
Parent
Re: (Score:3, Interesting)
Re: (Score:3, Funny)
Separate SVN deploys (Score:4, Informative)
Re: (Score:3, Informative)
Do _not_ use Subversion for this. Use git, even if you have to use gitsvn to point it to an upstream Subversion repository. Subversion's security models in UNIX and Linux are exceptionally poor, and typically wind up storing passwords in clear-text without properly notifying you. (Now it notifies you before storing it, but usex it automatically.) Subversion also has very poor handling of multiple upstream repositories, and there is no way to store local changes locally, for testing or branching purposes, an
Re:Separate SVN deploys (Score:5, Insightful)
Parent
Re: (Score:3, Informative)
What do you mean, git doesn't integrate with Apache? It works well as an Apache client, there's 'viewgit' if you need a bare web GUI. And for this purpose, locally recordable changes seems critical.
Re: (Score:3, Informative)
http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt [kernel.org]
trac does have git support too, etcetera
Re:Separate SVN deploys (Score:5, Informative)
Auth token caching can be easily disabled and svn export, not svn checkout, should be used for deploying test/prod environments (like I've seen way too many people do).
Git (or any other distributed version control system) is great if you are into distributed development, but don't blame the tool when you don't know how to use it properly or expect it to be something that it's not.
Parent
Re: (Score:3, Informative)
You wrote:
> You can say exactly the same about any form of password caching. You don't use Firefox because it can store your passwords?
No, I can't say exactly the same thing. Subversion, by default on UNIX and Linux, stores the passwords in cleartext. That is one of the stupidest things I've ever seen for allegedly "enterprise" class software. The only other source control I've seen do something that stupid is CVS, which is what Subversion is descended from.
"Configure in your dev environement" means that
Re:Separate SVN deploys (Score:4, Informative)
svn+ssh doesn't store anything in clear text. If that's a security concern for you, there's already a solution in place. Git is not the be-all and end-all solution to source control; it does many things very well, but there are a few things it does very poorly (repository control; with git, developers have a local copy of the repository which means that a stolen laptop comes with complete revision history). When the systems you're working on have certain sensitivities (legal, patent, security, etc), this can be a major weakness.
We do something very similar to the original submitter at work. We have 10-15 project branches open at a time. For us, we make sure that our code is subdirectory-agnostic (meaning it can run on the root of the website, or it can run out of a subdirectory). We use directory paths for branches, and we use internal DNS records for environments. http://de-appname/branchname [de-appname] would be a development branch while http://va-appname/branchname [va-appname] would be a validation branch.
For our lifecycle, we have development, validation, staging, pre production, and production. Development and validation are the only branching locations; staging, pre, and production are each single-path locations (though staging is a branch reserved for this purpose, pre production and production are /trunk)
On http://de-appname/ [de-appname] and http://va-appname/ [va-appname] there is essentially a directory listing along with the fully qualified branch name, revision, most recent contributor's name (even spelled out by looking up their record in LDAP), commit time, and most recent log message. Developers get a drop-down menu next to project branches which they can use to update the working copy there on that shared server (does a little ajax call and shows you the result in real time as though you were at a terminal). You can also create clean checkouts and even create new branches (either off of trunk or off of another branch). Finally you can even close a branch through this interface; it deletes the branch with a meaningful log message, and cleans up the files on that server. All through a web interface, no need to remotely log into a machine for this purpose. There's no reason for someone to administer this, each developer creates a working copy when and where he thinks it makes sense for himself.
Because our back end is SAP, we don't have to deal with multiple database environments. There's no "create a new copy of SAP" - indeed when this is something that's organizationally important (testing a major upgrade), it's a multiple day long process. We have a fixed set of data environments, and these are tied to hostname (going back to de-appname, va-appname, stage-appname, etc). If there were multiple database environments to worry about (eg, you wanted to be able to effectively branch database environments too), it wouldn't be a huge deal to set up a template database and have the same scripts we use to manage branches through a web API clone that database and update config in the appropriate app.
The key that I'm trying to get at is that you should create a web based tool to allow developers to manage this themselves. The developers will thank you because they'll be able to get what they want faster than you could have provided it, and they'll have control over when where and why working copies of their work in progress appear.
Parent
Re: (Score:3, Informative)
There are so many things to do to get this right - basically I'd add that you need a three-silo people model to match the three silos of dev,test,prod. Your development side are the creative ones; give them the tools they ask for and let them play. You need a critical, intelligent and demanding test manager in the middle, and for the production gatekeeper you need someone with absolutely no imagination at all (follow the rules, tick the boxes, or *zero* chance of advance to production. Seriously. Tell
happy with phing (Score:3, Informative)
There's really only so much you can do generically. I'm really happy with phing. I use the dbdeploy task to keep my databases in a similar state. I build on a local machine, deploy via ssh and then migrate the database.
I'd suggest that rather than checkout at each level you create a continous integration machine using something like cruise control or bamboo, then push out build tarballs and migrate the database.
Have the hosts email problems to an email account (Score:2)
When I did this years ago, each server would run scripts to read logs, etc and if they found something bad they would email me with what they found.
Simple and scalable
Re:Have the hosts email problems to an email accou (Score:4, Informative)
Never heard of a loghost eh?
Parent
Re: (Score:2)
Ever heard of Unixware!
remember this WAS 10 years ago!
(Now, get off my lawn!)
Re: (Score:2)
That's interesting. At the moment we have a loghost, and all logs of all applications go to that syslog server. Now we face the problem of allowing access to those logs to developers. Say you have 50 production apps logging to that logserver, do you know some software (best would be a webapp) that can be configured to let developers login and see the logs for the application they are responsible for? We could simply share the log files with a samba share, but a webapp that has some kind of integrated tail,
I thought... (Score:2)
/sarcasm (in case you couldn't tell)
Start an OSS Project (Score:2)
I've never actually done this (my employer balks at the suggestion), but I'd love to have that sort of opportunity.
SVN etc. (Score:3, Informative)
Still not figured out an efficient way to version MSSQL and MySQL databases using OSS, though. Open to suggestions!
Re: (Score:2)
My company wrote a small project for this (not released in any form, though). It has a collection of SQL scripts identified by date (eg "2009-10-15 1415 Renamed Foo.Bar to Foo.Baz.sql") and a table with columns for script name and date applied. Any scripts it finds that aren't listed in that table, it applies in order according to the date in the script name.
You should be able to hack this together in a day or so.
Re: (Score:3, Informative)
Deployment to live servers via SVN checkout when the time comes
Side note: I humbly suggest (as someone else mentioned elsewhere) you use export instead of checkout for the live deployments.
Leverage your issue tracking and cvs (Score:3, Interesting)
If you're able to script deployments from a configuration management host you can script against your CVS (SVN, SourceSafe, whatever-you're-using).
There are a lot of ways to automate the management of what file version is in each environment but a smart choice is to tie things to an issue tracking system. My company uses MKS (http://mks.com) but BugTracker or BugZilla will do just as well.
Your scripted interface can check-out/export the specified version from controlled source and FTP/SFTP/XCOPY/whatever to the specified destination environment. For issue-tracker backed systems you can even have this processes driven by issue-id to automatically select the correct version based on issues to be elevated. Additionally, the closing task for the elevation process can then update the issue tracking system as needed.
Many issue tracking systems will allow you to integrate your source management and deployment management tools. It's a beautiful thing when you get it set up.
Hilarity (Score:2, Interesting)
Another option is to completely rewrite the scripts (or hire someone to do it for me), but I would much rather use something OSS so I can give back to the community. How have fellow slashdotters managed this process, what systems/scripts have you used, and what advice do you have?"
I'm sure you have a legitimate problem, and there are lots of ways to solve it, but this line just cracks me up. You COULD write it yourself or pay someone but if you use someone else's Open Source work (note: nothing is said about contributing to an OSS project, just using it) you'd be "giving back to the community.
Translation: I have a problem, and I don't want to spend any of my own time or money to solve it, so I'm going to try and butter up the people on Slashdot in hope of taking advantage of the
Re:Mod Parent Up! (Score:4, Insightful)
Where eln failed is in how his post turned into nothing more than a personal attack against the parent story poster. In the open source world there are users, documenters, developers and visionaries. And guess what - a majority of those are users and most users will never contribute to your project.
Simply attacking the guy with crass, harsh statements is not in the vein of "The Gift Culture."
So, yes, eln's comment is a troll comment.
As for a "moral obligation?" That's laughable. If you give someone something for free don't expect them to do anything for you. Maybe that person doesn't have the time to invest in giving back at the moment. Making inflammatory comments will certainly push them away from your base of constituents. And that means less users. So attacking people who don't know is counter-productive and does not serve the OSS causes or beliefs.
If the someone feels to compelled to "give" or "give back" to the open source community - in whatever manner - count the community fortunate. Expecting anything is counter to the ideals of "The Gift Culture."
Please reread ESR's book.
Parent
I manage them with an Iron Fist Of Death (Score:2)
Or I would if I were in management. For some reason they won't promote me here.
Most important thing in my book (Score:5, Interesting)
Most important thing is to treat your code and data separately.
Code:
Dev -> Test -> Production
Data:
Production -> Test -> Dev
Many developers forget to test and develop with real and current data, allowing problems to slip further downstream than they should.
And make sure you backup you Dev code and you Production Data.
Re: (Score:2)
Dang. Out of mod points, so I'll reply.
Parent covers an EXCELLENT point. We've gone to great lengths to replicate data from production to test/dev modes. We have scripts set up so that in just a few commands, we can replicate data from production to test/dev, and that do data checks to make sure that something stupid isn't done. (EG: copying a customer's data from test -> production and wiping out current data with something 2 weeks old, etc)
In our case, each customer has their own database, and their ow
Re:Most important thing in my book (Score:4, Insightful)
Testing with real data is not necessarily a good practice. Consider sensitive data, such as social security numbers. Auditors may ding your development practices for providing developers access to information they do not need. You need realistic data, not necessarily the real data. If you're bringing real data from prod back to test and dev, consider having something scrub the data.
Parent
Puppet (Score:2, Informative)
Tools, Practices and Standards (Score:5, Informative)
We utilize a number of tools depending on the site, but generally:
Version Control (Subversion) for management of the code base (PHP, CSS, HTML, Ruby, PERL,...) - http://subversion.tigris.org/ [tigris.org]
BCFG2 for management of the system(s) patches and configurations (Uses svn for managing the files) - http://trac.mcs.anl.gov/projects/bcfg2 [anl.gov]
Capistrano/Webistrano for deployment (Webistrano is a nice GUI to capistrano - http://www.capify.org/ [capify.org] / http://labs.peritor.com/webistrano [peritor.com]
However, all of the tools above mean nothing without defining very good standards and practices for your organization. Only you and your organization can figure those out...
Check out Springloops (Score:4, Informative)
It's hosted Subversion, with a slick web interface that walks you through darn near everything. You can configure development / test / production servers that can be accessed via FTP or SFTP and deploy new builds to any of them with just a couple of clicks. It integrates with Basecamp for project management, and it is really cheap - it sounds like either their Garden or Field plans would meet your needs, and they're both under $50/month.
Check them out here. [springloops.com]
Not affiliated with them in any way, other than as a satisfied customer.
Look at Capistrano, steal ideas from Rails (Score:5, Informative)
Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.
Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.
You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.
I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.
best practice (Score:2)
The trick then is to move on to another outfit just before it hits the fan. Don't worry about your customers - if they are running w
What's a DEV environment? =:O (Score:2)
Did he forget the Sarcasm Mark ~, or does he not know about it?
TPS reports (Score:2)
TPS reports with lots of cover letters.
Packaging Packaging Packaging... (Score:5, Informative)
Its amazing, how this seemingly obvious question, always gets weird and overly complex answers.
Think about how every unix os handles this. Packaging!
Without getting into a flame war about the merits of any packaging systems:
- Use your native distributions packaging system.
- Create a naming convention for pkgs ( ie, web-fronted-php-1.2.4, web-prod-configs-1.27 )
- Use meta-packages ( packages, whose only purpose is to list out what makes out a complete systems )
- Make the developers package their software, or write scripts for them to do so easily ( this is a lot easier than it seems )
- Put your packages in different repositories ( dev for dev servers, stg for staging systems,qa for qa systems , prod for production, etc et c
- Use other system management tools to deploy said packages ( either your native package manager, or puppet, cfgengine, func, sshcmd scripts, etc )
And the pluses? you always know absolutely whats running on your system. You can always reproduce and clone a systems.
It takes discipline, but this is how its done in large environments.
-
Re: (Score:3, Informative)
In a word... (Score:2)
Fabric.
http://www.gjcp.net/articles/fabric/ [gjcp.net]
Saves so much hassle and buggering about.
How do i manage those environments? (Score:2)
Use CVS if you're feeling burly (Score:2)
CVS (of whatever flavor) can help you do this. It's a pain in the ass, and everybody will hate it, but it works.
I've done this with virtual machines as well. It's kinda whizzy to do, but probably overkill.
The simplest way for me was to simply use rsync. Rigid delineation between live and test/dev environments is important. Use a completely separate database (not just a different schema), and if possible a completely separate database server. Changes to the database schema should be encapsulated in updat
KVM/Vmware/OpenSolaris zfs go virtual (Score:2)
The simple answer? Virtual Machines. If you have to stay with linux, go with vmware or for a free solution, KVM. See http://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
If you want to run LAMP on open solaris/solaris, ZFS has very robust and easy to manage virtual machines called zones. Sun also provides enterprise ops center software that can be used to manage the zones via a gui. Copy/create/rollback, etc..
After that, smart system administration is required to keep things easy to manage.
How you
Quick Brief (Score:5, Informative)
Develop 4 Environment Structures
Development (DEV)
Integration Testing (INTEG)
Acceptance (ACPT)
Production (PROD)
For each system create a migration script that generically does the following:
(We will use SOURCE and DEST for environments. You migrate from DEV->INTEG->ACPT->PROD)
The migration script as it's core does the following:
1) STOP Existing Services and Databases (SOURCE and DEST)
2) BUILD your deployment package from SOURCE (This means finalizing commits to an SVN, Creating a dump of SOURCE databases etc.) If this is a long process then you can leave the DEST running and STOP DEST at the end of the build phase. I do this as builds for my world can take 2-3 days.
3) CONDITION your deployment package to be configured for DEST environment (simple find and replace scripts to correct database names, IP address, etc. These should be config files that are read and processes.) This is common if there are different security SAPs, Certificates, Etc that need to be configured. For instance you may not have SSL enabled in DEV but you might in INTEG or ACPT.
4) BACKUP DEST information as an install package(this is identical to the BUILD done on the source. This BACKUP can be deployed to restore the previous version.) This should be the same function you ran on SOURCE with a different destination (say Backups verus Deploys)
5) MIGRATE the install package from SOURCE to DEST
START DEST
6) TEST TEST and RETEST
7) If all tests pass then APPROVE. This is the green light to re-start the SOURCE services so development can move on.
That is a brief of my suggestion.
DEV is obvious ... yeah...
INTEG is where you look for defects and resolve defects. Primary testing.
ACPT is where user and BL acceptance testing occurs and should mirror PROD in services available.
PROD
I handle about 790+ applications across 2000+ pieces of hardware so this may appear to be overkill for some but it can be as simple as 4 running instances on a single box with a /DEV/ /IT/ /ACPT/ /PROD/
Directory structure with MYSQL running 4 different databases. The "Script" could be as simple as dropping the DEST database and copying the SOURCE database with a new name. Other options are creating modification SQLS for instance that are applied onto the exist database.
e.g. STOP, UPDATE, START
to preserve existing data. In the case of Drupal your DEV might pull a nightly build and kick out a weekly IT, a biweekly ACPT, and a monthly PROD update.
JUST REMEMBER THAT YOU MUST MAKE SURE THE PROCESS IS ALWAYS REVERSABLE!!
The script to deploy needs to handle failure. There has to be a good backout.
You should have a method to backup and restore the current state. Integrate that into the script. Always backup Before you do changes and AGAIN after you change. DEV may need to look at the failed deploy data (perhaps a substitution or patch failed, they need to find out why.)
Before Backup and After Backup in the migration script.
And always 'shake out' a deployment in each environment level to make sure problems to propogate. You find problems in IT, you test to make sure what you found in IT is resolved in ACPT. Your testers should NOT normally be finding and filing new defects in ACPT environments with the exception of inter-application communication that might not be available in earlier environments. (Great example might be ACPT has the ability to connect to say a marketing companies databases where you use dummy databases in IT and DEV.) 80/20 is the norm for IT/ACPT that I see.
Good luck. Use scripts that are consistent and invest in a good migration method. It works great for mainframes and works great in the distributed world too.
A special condition is needed for final production as you may need temporary redirects to be applied for online services (commonly called Gone Fishing pages or Under Construction Redirects)
From professional experience: (Score:3, Informative)
I have adapted my system from the 5 years I professionally did it.
First of all, it's a 3-stage system.
You have a couple of live servers, a identical staging server, and the user machines.
Every system has a clone of the files. the servers have rsync copies of the stage server files.
And the users all sync to the stage with GIT.
Everyone has a local clone of the stage server software too, so he can test server-side code right on his machine.
That's important in every company where people could do conflicting (and even big, global) patches.
The stage server then has validity tests running. Compilations and unit test cases wherever possible. Including the database, the server side code, and rendering test pages in all relevant browsers, to diff the rendered versions (images) of the pages. (There's a app for that in Firefox, but otherwise it's desktop automation.)
There's an red alert box in the test case overview when something fails. Which gets checked every evening, before pushing anything to the live servers at night.
The only thing that turns out to be a bit hard, is to test the client-side logic (e.g. of web-apps) in a transparent manner (= keeping the software configurations and serveride code the same to be able to rely on it).
Then there is a emergency push and and emergency direct live update mechanism, for cases when you quickly have to fix something that got overlooked. (Which usually should result in a new test case to be written, to catch all such problems.)
A well-integrated project management system is very important. At the end of my first company, it was a self-written one with good integration. But in the beginning, something like Trac might suffice.
Then very important is, to have a knowledge base for all things that need to be remembered. Like a meta-documentation. Workflows and procedures. Why the mysql server will not restart on a reboot of stage server clones. Little hooks and mantraps like that. I recommend a Wiki.
And last but not least, never ever forget to have a Bugzilla. If you're good, you can integrate Bugzilla, the test validations and the task/project management into one system. Making the validity tests create bugs in Bugzilla, and bugs being the same as tasks (which makes test-driven development easier).
Yet this all is completely worthless, if your colleagues don't use it! ;) ...". It's the mother of all PHB stories. ^^
Unfortunately, I learned, that when someone can/em> do something wrong, he will.
So if you can't lock down possibilities to only those required, you have to be very very careful with who you hire. Especially with "web development", where you get sinology students who learned HTML while working as a taxi driver, stating that they are "professional web developers with 5 years of experience", while honestly believing that. And team leaders believing it too, because they are just as "competent". Because they themselves either started as something an simple as link collectors, or the boss of the company does not know shit about his business, and hired those types. They then usually get promoted to "Head of
The key is: Make them like to work the proper way. If nothing helps, money can always push them in the right direction. It's called "bonus". :)
And making it their project too, by also embracing their decisions!
Re: (Score:3, Interesting)
I'm not aware of any systems that do all this for you while still being flexible enough to accommodate lots of unique requirements. The script-based systems I've employed over the years all followed some basic rules: