Testing Network Changes When No Test Labs Exist? 164
vvaduva writes "The ugly truth is that many network guys secretly work on production equipment all the time, or test things on production networks when they face impossible deadlines. Management often expects us to get a job done but refuse to provide funds for expensive lab equipment, test circuits and for reasonable time to get testing done before moving equipment or configs into production. How do most of you handle such situations, and what recommendation do you have for creating a network test lab on the cheap, especially when core network devices are vendor-centric, like Cisco?"
Pretty simple, really (Score:1, Insightful)
Whenever you're working in/on a production environment, only one rule matters:
Don't fuck it up.
Re:Pretty simple, really (Score:5, Funny)
Oh, no. We do this all the time. Around the holidays we rewire the production server racks so their ethernet cables droop over the aisles, so we can hang up Christmas cards. Jimmy has a script that blinks the blue UID lights for a festive holiday display.
Re: (Score:2)
You mean the IPMI LEDs? (Dell Poweredge servers have a dual-color LED that can flash (blue or orange) that signifies errors etc. Accessible via IPMI (along with all sorts of other goodies, like serial over Ethernet at a level higher than the OS))
Re: (Score:3)
Oh, a variation on blinkenlights?
ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKSEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
Re: (Score:2)
damn straight, thats why you get paid.
in theory, theory and practice are the same, in practice its not. you're job is to make it that way.
replace theory with lab and you see the fundamental flaw with the false sense of security a lab provides.
Re: (Score:2)
Holy blast from the past.
Re: (Score:2)
A very wise Network Admin once told me.
Wisdom comes from experience. Experience comes from mistakes.
So, you see most of us wise administrators got our experience somewhere else, if you know what I mean.
wink wink, nudge, nudge.
Re: (Score:2)
It's called a FOA first office application. You do what modeling you can, check what you're changing and Rule #1 is dont fuck with something if you know nothing about it. We do it in the middle of the night and if it screws up things we just restore the changed equipment to the pre change state. Networks are too complex and even the best lab modeling does not catch all situations.
Re: (Score:2)
Either that, or redundancy, redundancy, redundancy. I always at least try to convince the bosses that hardware needs to be ordered in even numbers- so that we have onsite emergency replacements.
That extra hardware can then be used to build test beds.
Re: (Score:3, Insightful)
Everyone has a test environment. But not everyone has a production environment.
The tag says it all (Score:5, Insightful)
There are zero replies and the story is already tagged with "youreboned". That's the truth. If your higher ups won't front the money for proper test equipment and expect you to roll out production-ready equipment on the first go, then you really are boned. Of course, you can mitigate this by simple pen-and-paper analysis. What should each piece of equipment do? Are the products we've selected appropriate for the roles we're going to put them in? These sorts of questions can find a lot of bugs without any sort of testing. If you think, "what would I do if it was the 1980's?" then you'll be fine.
Comment removed (Score:5, Insightful)
Re:The tag says it all (Score:5, Insightful)
Not all changes are a one-way trip. Having a rollback plan is also important. Should something very unexpected happen, be prepared to roll back any and all changes to undo what has just been done.
Couldn't agree more, except to say, don't assume you'll be rolling back from a known state. I've seen roll-back plans that assume they're undoing the changes just put in, not reverting to the state before the changes. Yes, there's a difference between the two! Eg, if your install fails, maybe you can't un-install. Yes, this might mean additional resources and the overhead of FS and DB snapshots, and complete copies of config files, but better that than the alternative.
Re:The tag says it all (Score:5, Insightful)
My comments are that out of band management are the networking guys best friend, and POTS is the best OOB available. Also learn how to change the running config without affecting the saved config, that way worst case is you have to power cycle (can be done with the correct OOB config or you can pre-schedule a reboot that you cancel if everything goes well). Oh and downtime windows might seem like a luxury but unless you are Google or Amazon the business needs to be made aware that they are necessary and critical to the smooth functioning of their IT infrastructure, so you should be making these changes during the downtime window where everyone is aware that things might break.
Re: (Score:3, Informative)
You bring up a good point regarding changing the running config vs the saved config.
What I'll do if I'm changing a remote system - POTS or no - is set up a reboot of the device in 15 minutes. After verifying the clock. Then, if something in the config causes an unforseen issue, you just need to wait a little for the switch/router to come back online with it's original config.
Obviously, this can extend the outage window - however, always plan for worst case...
Re: (Score:3)
Re: (Score:1)
Re: (Score:2, Informative)
If you truly believe that a simple reversion of a configuration will cause a reversion to a previous state, you're sorely mistaken.
Once the device you're working on starts misbehaving, other devices around it will start misbehaving too. As an example, one change to a network I'm involved with was supposed to simply prioritise VoIP traffic for one customer. The change was successful, the engineer went home. Then three hours later a major network router failed, because the higher priority voice traffic which
Re: (Score:2)
Re: (Score:2, Informative)
Not Pushing Juniper gear, but their Commit functions in JUNOS, and commands like "rollback" are serious things to consider in these scenarios. JUNOS also does things like refusing to perform a commit if you've done something obviously stupid (it does basic checking of your config when you commit).
Label me a shill. Whatever. JUNOS is a lot better from an operator POV.
Re: (Score:2, Informative)
My personal favorite thing about JunOS is "commit confirmed 10"
This can be a lifesaver, if you fat fingered something, and you break even your ability to access to the device, your transaction should roll back in 10 minutes.
If nothing goes wrong, you have 9 minutes to do some simple sanity checks, make sure your LAN is still working, and then get back to your CLI session and confirm the change.
Re: (Score:2, Informative)
On a cisco, you can just do "reload in 10" and "reload cancel". If you don't know about those commands, you really shouldn't be working on a production network unsupervised.
As for the original question: Either use similar low end equipment, or use your spares. (please say you keep spare parts around)
Re: (Score:2, Interesting)
"reload in 10" on a core router or switch (eg a massive switch that also has routing duties) is insane, and will probably impact the entire network, for 20-30 minutes, if you accidentally lock yourself out (but don't otherwise impact anything) and fail to cancel that reload.
In addition, reload is risky, and the equipment may fail to come back up correctly.
Sorry, it's not anywhere close to comparable to the configuration management features in JunOS.
"Reload in X" is a bad answer, and should never be
Re: (Score:2)
No, there's nothing wrong with that (Score:2)
As long as the downtime that will result is acceptable.
Re:The tag says it all (Score:4, Interesting)
Back around '01, I was working for First Union (which later became Wachovia). They had this massive corporate push for anyone and everyone in IT to roll out a standardized Software Configuration Management [wikipedia.org], and of course we were included. The big problem was the lab. The corporate standard was to test changes in a lab environment and then move to production (duh).
For a telephony environment, we had a pretty good lab that could duplicate most of our production scenarios, but not all. Another problem was there were a LOT of people with their fingers in the lab since so many groups were involved: eg. The IVR team is in there because you have to have IVRs in the system. Same with call routing, call recording, desktop software, Q&A, etc.etc.
So the lab was in a constant state of flux with multiple products, multiple teams, and different software cycles and endless testing always occurring. We made it work by testing the stuff we weren't sure about in the lab, only doing changes in prod after hours, and having really good testing and back-out plans.
So when the corporate overlords started telling use we couldn't make any changes to production without running everything through the lab first, we basically laughed and told them we'd need around 500 million for the lab and dedicated resources to run it. I ended up telling them that to duplicate the production environment, we'd need another bank as our "test bank", and we could test changes on the test bank and then put them in the production bank.
As with so many things in that IT department, it went from being a priority to fading away when something else became a priority.
Re: (Score:2)
Re: (Score:2)
But that's not the point. You would never use disaster recovery centers or redundant servers as a make-shift lab. If there was a problem on the production box and the redundant server wasn't available, you'd definitely be in deep doo doo.
My original point is if you have a
Re: (Score:2)
Could be worse (Score:4, Insightful)
The best bet is to be ready to blame the vendor when things go south ;-)
Seriously, I'm right there with you. If management does not want to provide for a test lab & reasonable time to test. Then it's clear they've made a 'business decision' that the network is not of sufficient value / risk is not great enough for such investments.
This may change quickly once something goes south (assuming they understand why it did) but you're gonna be talking to a brick wall until then.
It could be worse, you could have management that are afraid of there own shadows & who freak out at the idea of replacing redundant components after a HW failure. (Ever had to get VP approval to replace a failed GBIC? Oh, I have & yes, I hate my life).
Re: (Score:2, Interesting)
See how much approval you have to get when the network is down because of a failed GBIC.
Redundancies against component failure are very good for the enterprise, but also make it harder for engineers to do their job, since "nobody notices that something has gone wrong".
Perhaps the real redundancies should be reserved for the absolute most business-critical things.
Make sure less important things are non-redundanct and arranged in a way, so that if any link or GBIC does fail, something noticeable to
Re: (Score:3, Insightful)
That reminds me of an article by Nelson Repenning, "Nobody ever gets credit for fixing problems that never happened". It's quite an interesting read... The guy who "saves the day" during an emergency always seems to get credit and reward, but what about the guy who keeps the emergency from ever happening?
Re: (Score:2)
Hey! Thanks for that!!!!!
I'd heard variations on it several times but assumed it was just folklore or [un]conventional wisdom. Your post prompted me to search and find the article you mentioned: here'
Re: (Score:2)
Re: (Score:2)
Can someone please explain this sports analogy with a car analogy so I can understand it?
Which is better, Toyota (who's cars never fail) or the local mechanic (who's a whiz at instantly diagnosing and fixing Chevys)?
Re: (Score:2)
I think the point is that Willie Mays and Joe DiMaggio would both get the job done, but do it in different ways. Willie Mays would make the catches while running, diving, over-his-head, etc. All of which looked amazing and made the highlight reels. DiMaggio, did a better job of preparing and positioned himself better so that when catching a similar hit, he would be in position to catch the ball standing up. Both were great athletes, but DiMaggio made the job look routine. Even though he was equally capa
Re: (Score:2)
"Both were great athletes, but DiMaggio made the job look routine. Even though he was equally capable of making the fantastic plays, his approach was more safe because he didn't have to make them so often."
Based on this criteria, DiMaggio was a superior fielder. It would be better to have a DiMaggio running your infrastructure than a Mays. Other departments may have different criteria (say sales).
Excellent people make hard tasks look easy. Mediocre people make easy tasks look hard.
Virtualization? (Score:5, Interesting)
You can also use QEMU to create virtual network nodes. If you have enough RAM, then this can help at least get the logical issues worked out and the software configurations square. Then you just need to do the real work
Re: (Score:2, Informative)
Re: (Score:2)
Re:Virtualization? (Score:5, Informative)
Specifically, GNS3 allows you to create test networks in a virtual environment, then import software images for your Cisco routers, switches, PIX firewalls, Juniper hardware, etc, all run on hypervisor technology.
For anyone unfamiliar with GNS3, a link to the website [gns3.net]. There are versions available for Windows, Linux, and OS X. FreeBSD already has it in ports.
As a side note, I'd add that maintaining a home lab (to the extent practicable and useful) is one way to side-step limitations of what your employer provides. Consider it a combination of "Ongoing Professional Education" and "Proactive Job Security Measures" (i.e., "I better test this shit to save my ass tomorrow").
Re: (Score:3, Informative)
If you work a pure Cisco environment, talk to your Cisco guy about getting Packet Tracer. Emulates a few routers and a lot of switches. It works really well. Plus, 5.1 adds virtual networking. You can design several networks on several laptops and then join those networks over a virtual internet.
You could simulate the net with Packet Tracer (Score:2)
Granted, it's not really an ideal solution, but it may wind up being the only way to avoid using production equipment.
Document and test at night (Score:5, Informative)
Re: (Score:2, Insightful)
If you get fired for failing to do a job for which you were not equipped (and they know you aren't equipped for it), you might be able to sue because they created a hostile work environment. Hostile work environment lawsuits aren't just for sexual harassment, folks.
Re: (Score:2)
There's a potential hitch or two in your plan.
If it goes smoothly anyway, you might look like a whiner that didn't need the expensive toys to keep on the shelf. They feel vindicated. If it goes poorly they'll assume you didn't really try because you wanted to prove yourself right.
Re:Document and test at night (Score:5, Funny)
Hence, you have the plug to the main router beneath your own desk. When the sailing looks smooth, you kick out the cord. While everyone freaks out, you open up a terminal window and begin typing nonsensical commands. Say, "Ahaaah! As you re-plug in the router.
Job security.
Seth
Re: (Score:1)
Sounds like Bastard Operator From Hell to me. But it could be the only defense against Incredibly Incompetent Manager From Hell.
Re: (Score:1)
I use fire.
Re: (Score:2)
Re: (Score:2)
With your feet? You ARE talented!
Re: (Score:1)
Re: (Score:2)
Re:Document and test at night (Score:4, Informative)
step 3) If possible test network changes on the production equipment at 2am so that impact on users will be less
Been there, done that. Sadly the only way to see how your setup works is to try it in production.
Sure it helps if you can test it beforehand, but sometimes your lab might not reflect what happens in real network when you roll something out.
Just make sure you can clock those am hours as overtime/nighttime work.
And remember to backup the running config twice so you can restore the production network if something goes fubar.
Re: (Score:2)
Been there, done that. Sadly the only way to see how your setup works is to try it in production.
The other thing to mention is to be honest with the other technical staff about if you've actually made a change, even if "trivial". This is because sometimes when you modify something, you can end up dumping them in the shitter accidentally, e.g., by putting a critical service on the wrong side of an internal firewall so that no packets get routed to it at all. In fact, I saw that once and networking stonewalled for a week before admitting that indeed they had made a small modification "that shouldn't have
Re: (Score:3, Informative)
Re: (Score:2)
That means your experimental model is not good and needs to be refined.
You see - all those guys that did a six month course and call themselves "engineers" could have had some benefit of a real engineering education or the experience of working with real engineers.
Meanwhile I have idiots learning about routing or DHCP on production systems because they can't be bothe
Re: (Score:2)
Re: (Score:3, Interesting)
Note, step 4 may get you fired instead. Business decisions are somewhat nondeterministic.
And that's what happened to me.
I was forced into making changes in the production environment, and caused an outage that affected 2 people. Once I realized what happened, I quickly fixed it; however due to internal politics I was terminated the next day.
Initially I was in shock. 10 years, 2 months employed in a single company. Gone. I have a stay-at-home wife and 3 kids; which made things look even bleaker.
In hindsight, it may be one of the better things to happen to me. I had spoken with a recruiter a
Re: (Score:3, Interesting)
3) If possible test network changes on the production equipment at 2am so that impact on users will be less step
That's dangerous. You leave it apparently running and crawl back to sleep at 4:30AM, to get an angry call at 7:05AM when the first users to log in report something essential is fucked up.
Prepare and test at 2AM, then roll back to original. Then re-apply around lunch break and wait with your fingers on roll-back for the first reports of failure.
Re: (Score:2)
3) If possible test network changes on the production equipment at 2am so that impact on users will be less step
You're a network guy, right? How well do you know the applications that use your network? How sure are you that the application behind, or in front of the change you're making don't need a restart after losing connectivity? Maybe your late night tests are causing all sorts of problems and expense when the apps guys come in to find the system inexplicably down, having visible outages, and have to start raising support requests against vendors to find a solution to their non-reproducible high severity defect
Re: (Score:2)
Interesting analysis. I've observed just such a scenario, and the difficultly lies in product issues (yes, the product behaves this way... until the next upgrade) and the layer between the apps guys and the app (the support guys). In an ideal world I'd agree with you, but we're operating in a business world, so I'll modify my statement to say, don't mess with the network after-hours if it's not your apps being affected.
Re: (Score:2)
4) Once the inevitable failure occurs, haul out the paper trail and get the bean counter fired. Repeat until test lab is approved. Note, step 4 may get you fired instead. Business decisions are somewhat nondeterministic.
And this is the part that SUCKS.... A while back, I was part of a three-way integration project, with myself (representing a vendor), another vendor, and the ultimate customer. In advance, I'd talked through everything with the other vendor so we had a clear plan, including a verification st
My last resort (Score:5, Funny)
I call my buddies at RIM and test my mods on their system.
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
You can tell if you see a bunch of little black berries
Re: Testing Network Changes When No Test Labs Exi (Score:2, Interesting)
I would suggest asking your vendors for demo or evaluation equipment. Cisco, Juniper and 3Com have pools of demo equipment as do the resellers like PC Connection and CDW.
I've done deployments of new switching infrastructure based on work I've done with loaners from my vendors. It can be tough because the typical evaluation period is 30 days. Although you can get 45 and even 60 days.
If you have a good relationship with your sales rep. It would be easy to push them to get the necessary items to do basic testi
Packet Life (Score:3, Informative)
Tools (Score:5, Informative)
Here are a few tools:
GNS3 - http://www.gns3.net/ [gns3.net] - free network simulator, based on Dynamips Cisco emulator
Opnet - http://www.opnet.com/ [opnet.com] - detailed planning of networks, from scratch
Traffic Explorer - http://packetdesign.com/ [packetdesign.com] - plan changes to an existing network
Re: (Score:2)
Those are great recommendations, thanks!
lots of little things (Score:3, Informative)
Older Cisco equipment can function just as well as newer for 95% of lab scenarios. You are very unlikely to be needing to use all the newer features.
Anything that can run IOS 12.3 and is newer than a decade old can do a lot more than you think. We do all our BGP testing on a stack of 2600s and 3600s and never an issue even though in production its 2800s, 3800s etc.
Granted there are features that you do need the newer kit esp when syntax changes (e.g. IP SLA commands, newer netflow commands, class map based QoS to name three off the top of my head) but none of the core routing and switching features/commands has changed much since the introduction of CEF - they all do ACLs, route maps, OSPF, BGP, EIGRP, vlans, spanning tree, rapid spanning tree, IPSEC vpns. I'm speaking from an enterprise POV not a service provider but I'd imagine if you are in a telco environment you wouldn't be lacking gear.
For many minor test scenarios, you can pick a test branch office and use the good old 'reload in XYZ' command to ensure that no matter how badly you stuff it up, everything will bounce and come back (just remember NOT TO COPY RUN START lol).
Then there's the sleight of hand methods:
- always ordering more for projects than you really need. Par for the course really esp as most project managers haven't a clue when it comes to the nuts and bolts of a big cisco order.
- pushing for EOL replacements as early as possible, intentionally conflate end of sale with end of life.
- getting stuff in for projects as early as possible, then you have a month or two to use it as test gear.
- remember that your lab need not mirror reality, scale down as much as possible. e.g. to simulate a pair of 4506 multilayer switch running in VRRP, use a pair of 3560s. Use your CCO login and flash away to your hearts content (I know its breaching licencing but for test scenarios, meh).
Rancid? (Score:2)
Re: (Score:2)
Go virtual! (Score:3, Informative)
If you are unable to recycle old equipment into your testlab you should go virtual.
For Cisco routers, GSN3/Dynamips (www.gns3.net) is your friend. Any recent PC or laptop will allow you to build a large and complex topology that will satisfy most experiments and even support you when doing certification preparation. It will only work for routers so switch-based platforms are out (like the 3570,6500 and 7600). The good news is that the features are more or less the same and they more or less behave the same way. If "more or less" is not close enough you need a replica of your production network or at least a few devices of each to test what can be labelled as critical.
For Juniper routers, google juniper Olive. It will run a juniper router the same way dynamips runs a Cisco router.
In both cases a proactive partnership deal with the vendor will be a good idea. Both Cisco and Juniper (and I am sure all other major network vendors) have programs where they will more or less advise, test and prepare the configurations for you. If you run a critical network this is money well spent.
In the end it comes down to the level of risk your management is willing to take. Ask them if they will allow the network to be less up since you are unable to properly test your changes before implementation.
Out of hours changes, and change managment (Score:3, Informative)
For any sort of medium to large network, you can't fully simulate it. That means you're always going to be making "untested" environment. So, you make very few changes rather than lots, you make sure after each change they've had the desired effect, and you have backout plans.
Borrow a lab! (Score:4, Interesting)
Cisco have many (large) labs located around the world. Sign up for some time in one of them.
Plan, inform and be prepared! (Score:1, Insightful)
Been there, done that (A LOT!!)
But it has failed quite a few times too..
If no money available for test labs, make good plans... Tell the dudes that wanted the changes (or if you are the dude that wants the changes inform the correct people that you will be doing stuff) Agree on a service window. Have backup plans.. Have all configurations saved.. Let all users know that after 10pm on that saturday network will be down for 10 mins etc etc..
Have tons of contengency plans, and let the 'responsible' people know
The power of logic compels me! (Score:1, Interesting)
You do not mention that this has ever made shit hit a fan. I conclude that so far this has not occured.
Consequently, you have proved that you are able to work without expensive test equipment by a combination and motivation and elbow grease. Congratulations!
Now, what is the logic for someone with a finite pool of money to provide equipment for someone who obviously does not NEED it? Yes, None At All!
You can therefore:
1) Wait until shit hits a fan and say "well, that's what happens when we don't have test eq
Paper Trail (Score:4, Interesting)
Make sure that every change request implementation documents that this change is being placed intro the production environment for testing. Document impact ranging from total network failure to moderate inconvenience and include roll out time tables. The roll out needs include travel times such drive to site B or fly cross country.
Of course the downside of this is that management may go out and hire someone who knows, or at least pretends to know, how to drop changes into place without whining about ignorance and making customers uncomfortable.
Some pointers (Score:2)
It depends a lot on your environment and the complexity you are dealing with. Test labs are wonderful things, but typically you end up in a situation where your network is so limited that a lab won't help much, or your network simply too complex to create a sane lab environment without dedicated staff and a huge budget.
Building a full scale lab is a large undertaking. It takes time and effort. You will need taps (for routing information), traffic generators, topology management and more. In my experience it
Don't forget SOX (Score:3, Informative)
1) You should not be making any direct changes to the network with out correct design, test and sign off.
2) You should already have a redundant network structure, so "half" can be loss without any loss to network operations. This way the change can be tested in parallel.
3) You should always report to SOX officer when a request outside correct operations and management is made. It makes it their responsibility to solve the legal issues, for not following their written standards, before you began.
Re: (Score:2)
Not that this isn't a good idea for other reasons, but how exactly does this requirement flow from the Sarbanes-Oxley Act? I mean, the whole thing is about financial controls, accuracy, disclosure, and reporting. I suppose your network could impair the timeliness of some reports, but it is hard to see in general how it is going to affect their accuracy.
Re: (Score:2)
Access Control - using VLANs and "firewalls" can be broken with errors
Business Continuity - system is unavailable for *any* work
Damage - system needing to be reload to return to working status w/ loss of intermediate work.
Notification - Is there some thing that maybe wrong
Missed Posting - due to todays separated servers one being "down" could cause *lost* postings and revenue.
That is just a short list. SOX opens a whole array issues, Best to let the SOX team to worry.
The network is now the system!
Re: (Score:2)
If what you know about Sarbanes-Oxley comes from Slashdot, you probably think it is significantly wider in scope than it actually is. You're correct: Sarbanes-Oxley is only tangentially related to IT, but somehow Slashdot thinks that it's this big huge-ass problem. See here for more information: http://en.wikipedia.org/wiki/Information_technology_controls#IT_controls_and_the_Sarbanes-Oxley_Act_.28SOX.29 [wikipedia.org]
Specifically this:
"The 2007 SOX guidance from the PCAOB[1] and SEC[2] state that IT controls should only
its the time to (Score:1)
polish your resume.
Download vyatta (Score:2)
Download an iso from Vyatta and build a test network with old PCs and spare NICs for testing. Sure, it's not the exact same as Cisco, but if they're too cheap to buy the real thing for a test lab then you'll at least be somewhat close.
Then, once you realize what you're not getting for your money with Cisco, you can buyt $1000 1U servers and build your own routers (or buy them prebuilt from Vyatta for about $2000) to replace the ciscos and make a profit selling the used Ciscos on ebay.
I do NOT work for nor
Re: (Score:3, Funny)
Re: (Score:2)
Either is fine by me, as long as it keeps me from having to use Cisco.
Re: (Score:2)
I do, and I do it well. You'd be surprised how much setting things up on linux / vyatta is to a cisco router. BGP is BGP. Of course, if you're not smart enough to use the knowledge gained from testing on the free stuff, I understand. Not everyone is.
UNH-IOL (Score:1)
The UNH-IOL is a neutral, third-party laboratory dedicated to testing data networking technologies through industry collaboration.
http://www.iol.unh.edu/ [unh.edu]
Get it in writing, let it fail. (Score:2)
You answered your own question. (Score:2)
As you already said, we secretly test on production in such cases.
Don't do it (Score:2)
Management hates paying for double the equipment, but for any production environment, it should be the cost of doing business. It minimizes risk and provides hot spares faster than an HP (or whatever) tech shows up. You should get some duplicate hardware for staging.
If you can't do that, then refer to the earlier post [slashdot.org] - don't fsck up.
Re: (Score:2, Insightful)
Don't just blame the management... (Score:2)
...You're as guilty - if not MUCH more - than they are here....
Quoting you: "Management often expects us to get a job done but refuse to provide funds for expensive lab equipment"
Well, have you considered it might be that you may not have informed the management from the start what's to be expected in the future? If there is ONE THING that the management does well and knows better than most of us - is how to EARN and KEEP money, they trust YOU to do your job and know everything about it so it doesn't have t
Services (Score:2)
You hire Professional Services from a lab/test equipment manufacturer (Spirent, Ixia, BPS) or dedicated testing companies (EANTC or others). Most of them will accept to work during the night, so you need to get a "maintenance" window where they can inject traffic. I do that all the time, from the testers side. It's stupid to do, by the way, because you should always test *before* production.
But that's really dangerous and the best way is still to test in the "lab". A lab can be a temporary rack where you pu
A simple trick that might help in some instances (Score:2)
(tftp/scp/etc new-config to router)
router# reload in 2
router# copy flash://new-config run
(something along those lines, this is off the top of my head, basically copy your new config to the running config)
if it works, wr it to startup config, if you get disconnected, wait 2 minutes for the router to reboot and automatically load the previous startup-config. Adjust
Many think this is normal (Score:2)
It's highly distressing to encounter these people, but many, tech and manager alike, actually think there's nothing wrong with working on production systems. To them that's just how it's done. They know no other way. Trying to educate them is met with blank stares and sometimes even harsh resistance.
cheat, lie and steal! (Score:2)
seriously, buy a new router to replace a 'broken' one from a location and then somehow fix the broken one for your lab/office.
The truth is that sometimes you not only lack the equipment for lab testing, but also the real world usage scenario. I am often stuck in a situation where I must backup a config and then experiment with production equipment and so am forced to do this outside of business hours. I usually get a chance to do some functional testing offline but cant really put new systems through ther