Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
News

Do You Have Your 'Crisis Week'? 133

Posted by Cliff from the how-do-you-handle-stressful-situations dept.
pmbarth asks: "This week, the large company I work at is having a 'Crisis Week', where we simulate different types of problems, and have training on how to deal with them. Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'. I was wondering how many other companies out there are actually training their non-IT employees on how to be aware of, and perhaps even counteract these types of issues?" It's an interesting idea, and one can't tell when an extra skill one learns on the job may come in use in a critical situation. Do other companies have something similar? Do you think such drills are particularly effective?
This discussion has been archived. No new comments can be posted.

Do You Have Your 'Crisis Week'? More

Do You Have Your 'Crisis Week'?

Comments Filter:

  • Depends on the industry (Score:1)

    by Anonymous Coward
    From my experience, "modern" industries and businesses don't take the time to worry about disasters. They either think it won't happen to them, or they'll just naturally handle it.

    Older industries, particularly manufacturing, do this fairly regularly. Of course, manufacturing as an activity has been going on for many years, so they've learned the hard way.

    Power failure tests, system control failure tests (when your furnace burns itself up because you couldn't regulate it, and you spend 7M$ to rebuild it, I bet you work very hard not to make the same mistake in the future.)

    As for IT-attacks, I haven't encountered that, but it makes good sense.

    With all of the failure tests you should start with planned failure test, and eventually run "unplanned" (at least from the perspective of some of the employees) test to see how people respond. It's easy to remember where the emergency exits are in a maze of corridors and factory equipment when you know there's going to be a drill :)

  • We've never had one... (Score:1)

    by Anonymous Coward
    ... in any of the companies I've worked for but as a sysadmin, I've always had a policy of "self-regulation", staying late a couple of times a month and purposely taking servers out of service and seeing how long it takes me to get them up, restore files from backup, etc.

    That said, when I quit as sysadmin due to "political" reasons I went on holiday for a week and when I got back it took 30 seconds to work out a server had been compromised and was being used to launch DOS attacks on www.microsoft.com. My replacement had spent a week replacing everyones NIC blaming the bad network performance on a faulty network card.

  • Re:OT: Interesting (Score:1)

    by Anonymous Coward
    They should replace interesting with a dynamic value that users can moderate. The values should be:

    Incredible
    Interesting
    Mediocre
    Redundant
    Tiresome
    Completely Erronious
    Dont Bother

  • ...or at least generate email.
  • Is there any such week where IT doesn't get attacked? I thought that's why we got paid the big bucks? Outlook viruses, RIP left on on the firewall and gated started, bad firmware, cut fiber, BGP flaps, IIS worms, named worms, DOS attacks, need I go on?

    kashani
  • Gonna be kinda hard. There's only one Slashdot, and it's not sentient. Your revolution will have to wait a few years.
  • Could be illiterate journalists. We've seen enough of that situation, too.

  • It's when you pull the plug on your daily XFS CVS compilation and stress testing because someone finds it annoying that the machine reboots every night even if you plucked all the problems this was causing, namely the not-in-the-kernel-because-they-are-binary-only-cra p NVIDIA drivers weren't being recompiled along and the some people left some mp3 players open on a no-longer-existant NFS mount, and then they complain because now there are bugs showing up in that freaking old version of the kernel because the machine no longer rebooted (bugs which incidentally wouldn't have showed up if the recompiles had kept their pace) and someone claims they lost two whole freaking hours of valuable work because of this problem (which I'm sure wouldn't have happened if he hadn't heard of the problem in the first place) and then to calm people down you upgrade the kernel to a current version only to find that hardware thing that had been creeping on the box suddenly shows up in all its glory and some moron that thinks he knows (dick) about the problem because he reads LinuxToday starts giving his unrequested opinion about it. This while all you want to do is some real work. This is a crisis week, and it spans over 14 days and counting.

    No, seriously, rant aside, I'd love to have something like this. Maybe people would actually learn to differentiate between it doesn't work *whine* and this particular part of the infrastructure has this particular problem, where that part of infrastructure is something more specific than the network and the problem goes beyond it doesn't do what I want (I'd pay for "it worked ok until I did this"). I'd also love if people were able to spot a problem ("hmm... I type ls --weird-option and it doesn't recognize it anymore") and report it instead of thinking "oh, the planets must be in the wrong position, I'll try again next week" and do zilch about it.

    And while day dreaming, can people stop saying "could it be possible that foo and bar have a problem?" if a) they know there's a problem and b) they try to smooth it out because they know it pisses the hell out of me when I hear "foo and bar just don't work". If you are going to say that, say it without the sugar, please.

  • ... then my opinion woul d be that your people aren't practised enough. If people always reacted badly in crisis situations even with training, then why would the military bother training? After all, it doesn't get much more high-tension than seeing the person next to you get blown to bits...

    Granted, continuous training isn't always worth the time lost in it, but still, don't dismiss an idea like this out of hand just because.


  • I work for an architorture firm and we have crisis weeks every week. what we train for is the annual slow friday .. only got one chance cant' mess that up

  • Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.

    Not at all, in a lot of cases companies don't even plan for problems they know they will have. A friend works at a major estate agent here in the UK, when recently they announced large branch closures. Firstly they announced these to the TV stations before their own employees, with a couple of weeks notice before closure. Secondly they failed to come up with any plan regarding what to do with the properties where they managed shorthold renting on behalf of others. Customers were left frantically phoning branches trying to find out this information for days before a decision was actually made.

    If multi-billion pound companies can't prepare for obvious short term definite eventualities how can we expect them to have plans for remote eventualities.

    Another problem is who reports these things. At a previous (Fortune 50)employer 50+ programmers were left for almost an entire day without power because no-one had the initiative to call in the problem, I (the summer student) eventually sorted out the problem (I have the unfortunate curse of feeling responsible for fixing other's mistakes).

    On the subject of breakins, the same company managed to let theives get away with 60+ Sun Workstations and a room-sized MAINFRAME, with full security on watch at the time. How?, I dont know! As far as I can tell, gross incompetence of companies as wholes is rife.

  • Where I work, every week is crisis week.
    ---

  • I am having a crisis week where I work. When I came in on Monday my HD failed to spin up. Which was pretty bad considering it wasn't backed up.



    I guess its time to RAID my desktop and pray my FS doesn't crash, since backing it up really isn't an option considering the amount of space on it. I would need to spend 10x on a backup solution. Of course I do backup the 'critical' data onto the server and on zips.

  • And you better believe they're gonna let you know, often and loud.
  • When in trouble,
    When in doubt,
    Run in circles,
    Scream and shout.

    (Author unknown)

    Others, anyone?


    The abbreviated Laws of Thermodynamics:
    1)You can't win.
    2)You can't break even.
  • My two weeks paid vacation are crisis weeks here.
  • I'm a doctor in a hospital and we have "crisis" simulations all the time. IT crises are the least of our problems.

    Crises to plan for:
    "Boeing events" - ?400 injured -can the ambulance/ER/OR cope? How many doctors/nurses can you get to the hospital in 30 minutes on fridays night? I've seen this simulated with 50 patients in a rural hospital. They employed actors to test this, and used the results in to extrapolate to other hospitals. It was great fun, they didn't tell us it was a simulation until we arrived (talk about crying wolf). The ER was overflowing with actors in bandages pretending to die while a surgeon suggested to the study co-ordinator that he could amputate with a hammer since no other equipment was available... :)

    "Power cuts" Power goes and the backup generators fail. Not cool if your in theatre with someone's chest open. The aneathetic machines have good batteries, but the lights have only minutes of power to clamp/close important things.

    IT problems are a joke. When the do snapshots ie declare "at 0935 today all computers/phones (which means all xrays,labs results, communications) go, then survey what might have happened" the number of predicted deaths is relatively small. Sure its a major distruption, but not too many people die.

    Seriously these "crises" are real issue for places like a hospital. Disasters happen. People MUST plan for them. IT disaster just don't feature compared to earthquakes, big plan crashes, total power failure etc...

    Elvis
  • >They took away the coffeemaker?

    We're talking attacks, and you start WW3..

    //rdj
  • Ok, this may be off-colour, but I laughed my ass off! Thanx for a morning pick-me-up.

    --
    Later...
  • Working in tech support for an ISP, we have escalation procedures defined for just about every fire/network/telecom outage. It's called a service level interruption procedure, or SLIP for short. We "practice" each and every day :)
  • Fjord and Dancin Santa, I'm so happy to see such enlightenment! Most males, on mention of this particular "crisis week," attempt to suppress gagging and run.

  • No, its after "Sexual Harassment Week" right before "Voluntary Leave Week" and "Office Shooting Day".
  • We did something similar, mailed everyone a .VBS script that went into the registry and disabled their ability to run....VBS scripts.

    It worked well until we did an AV procedure on our Mail store and it efficently deleted them all ;-)

  • So of course, your post has been moderated as Interesting. :) You can win for losing, can you?

    --Ty

  • we have crisis fridays. Basically, all our customers realize that things which don't get done on friday won't get done till monday, so they freak out and start asking us for things to be done that same day, on friday. This creates wonderfully stressful situations.

    We even thought of not working on fridays but the plan never worked :)

  • Call it crisis week de chinois. Hackers from a certain eastern nation threatened to attack US government sites over the course of a week. The IT folks a certain government academy fell all over themselves trying to prepare for an onslaught and in the process accidentally disrupted outside connectivity for a day and brought down the students' mail server for several days over finals. Of course, there were no attacks from the outside, but we sure had a week of crisis.
  • I beleive that the only reason the other people in a company are hired is to alert the IT department when there is an email problem.
    =\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\ =\=\=\=\=\

  • Pah.. (Score:1)

    by Motor ( 104119 )
    Drill? Where I work, every week is a crisis week.
  • I wouldn't be surprised if these people would give out dangerous information to a clever social engineer.

    Yes, that is another issue you could make them aware of. Why not take a day, or even a few hours of a day to educate the emplyees in basic security awareness?

    "Do not give out this information to ppl outside the company. Do not click on these .exe & .vbs files that was attached to your email. Do not use Outlook express."... And so on...

    Having drills otoh... I dunno, how would you set them up?

    --

    "I'm surfin the dead zone
  • The same drill can also be applied to memory leaks. It does not work for core dumps.
  • A couple of the companies I've worked for do have "fire drills". However, they are not done on a given week, but individually for each employee.

    The list of things was fairly involved: rebuild the servers from backups and from scratch; switch to an alternate pipe (isdn, dialup) if the primary failed; run through the restart procedures on the critical systems (necessary because you couldn't just power them back up); plus various repair procedures for filesystems, hardware, etc..

    The rebuilding of the fileserver was particularly useful. In one case, we realized that though a system was emailing lots of "successful" messages, the backup was useless in recovering the system. I know restoring is the other half of backing up, but at this place, the job was so onerous that it was rarely actually performed.

  • great - illiterate sysadmins!

    they probably can't even read
    .oO0Oo.
  • well take a look at security focus

    www.securityfocus.com

    And see all the holes and exploits available for your system. That should be a start
    .oO0Oo.
  • "Oh, and internet access for the whole network is through a single 64K ISDN line."

    you are so spoiled. I ran an ISP with 50 users on a 64k line for 6 months

    The COUNTRY of Gambia has
    International Bandwidth (Kbps): 128

    from http://www3.sn.apc.org/africa/index.html

    I picked Gambia because it had 64k the last time I looked. They must be well pleased with the upgrade!


    .oO0Oo.
  • there probably wouldn't be any electricty anyway but..

    put your server in a lead lined case of you're that concerned. People need a Quaker server in times of nuclear war
    .oO0Oo.
  • I for one am tired of all these interesting stories. I demand boring stories on Slashdot!

    Be careful what you wish for, you just might... oh. Too late.

    (Score: -1 Flamebait)
    (Score: -1 True)
    (Score: -1 Very True)

  • We're doing an unscheduled major code realease to our website today. I think this qualifies as a "crisis week".

  • The sysadmin probably knows this already, unless he's busy getting somebody's coffee cup out of their CD-ROM drive. Stopping them to complain about it only delays action. :)
  • every single crisis didnt end up on my desk anyway. Unfortunately of the few people in our company, I am the only one with any technical know how it seems. If we did have a crisis day, it would just consist of me getting phone calls complaining that their email sends are going slow or they cant reach a website, just like an ordinary day. This is a crisis to these people.
    fear
  • I work in the retail sector. We have crisis weeks all the time. They're called "sales".

  • I know people who don't even keep backups!
  • I work for an ISP and some of our coustomers practice this. The latest IIS worm owned their SQL server. Their soulition? Remove and reinstall the SQL server and NOT do testing on the rest of the network. Their about to find out the real meaning of this topic. Sigh
  • yeah, but when they say "the Internet is down" it has a totally different meaning. THey think that some appocaypse happened somewhere and destroyed "The Internet"

    The slashdot 2 minute between postings limit:
    Pissing off hyper caffineated /.'ers since Spring 2001.

  • You have it backwards.

    The only reason you were hired is so people have somebody to alert when there is an e-mail problem.

    They were all hired to generate revenue.

  • We don't have time for a crisis week

    Everything is in a crisis as it is.

    Seriously, think of the fun a pointy haired manager would have with this!

    Check out the Vinny the Vampire [eplugz.com] comic strip

  • ...you mean that you actually have to set aside time for your crises? And that they only last a week?

    Need a Perl programmer?

    b&

  • I'll give a real crisis--I work in a Microsoft shop. Crisis is the damn COM+ $@&# giving me a bad memory error everytime I try to shut it down. When you have to reboot everytime you rebuild a web app to test it, that's serious downtime.
  • oh ya. Typing one charactar at a time with a pen. Have fun!!!

    Sanchi
  • Yeah. That was the week IT took away my Unix-based mail and made me switch to Microsoft Outlook. Now they warn me not to open attachments that could flim-flam Microsoft Sillyware.
  • Thanks for the link, I must stay on the lookout for "script kitties," scratch0ring my sofa :-)
  • I have my "Crisis Week." It's called, "finals." In 37 hours, 23 minutes, and 8 seconds, I am gonna be... so fucking drunk

    --
  • So, when they simulate the network hacking attack crisis, just run down the hall to the data center. Hit the big red switch on the wall.

    You will see that the network hacking attack exercise will be successfully ended, and everyone will begin the simulation of what to do when the data center loses power.

  • Right on, my brother. It's been a couple of years (gee, time flies) but I was trained that it was always better to know where to look things up then to try and know everything. STAN-EVAL just ate that shit up. I wish we could have done more with the continuity binders though, we were always so busy putting out fires and holding the office together we didn't have much time for things, like proper docs and recurring training.

    I was a 1W051 (Weather Observer) and every unit I was at always seemed to be running as fast as they could just to stay in one place. I was never able to make much progress trying to make the place better, it was very frustrating and is one of the reasons that I didn't reenlist.

    Of course I'm in the same boat with my current job, running as fast as I can just to keep from being swamped. The difference is that I have hope in my current job that it will get better sooner rather than later. Also in my current job I really can make a difference and make things better for myself and the others around me. Sometimes it helps being small.

  • ...and if it really is a child trapped under a vehicle, you'll be able to count on Lassie to let you know.

    Therefore, if it's not from Lassie, it's not a crisis.

  • I read this, and was amused.

    Then I got to the bottom of the page, and saw this quote:

    Your love life will be... interesting.

    Ya gotta love irony.

  • Whoops... They told me it was two weeks ago. They hired me to be an outside attacker. Apologize to the boss for me for the DDoS of his desktop machine. Oh, and compliment "The Captain" on his pron collection would ya? :-)

    --

  • Router goes down, half the company is staring at blank screens. And what's the biggest complaint I hear?

    I can't get to CNN/eBay/Am I Hot or Not/online bingo/every other non-business essential site.

    Of course, what's the first thing I do when the router goes down? Dial my laptop to a local ISP and make sure I can get my /. cause, dammit, this is important!

  • 1. CEO Fired [How fast can you find a new one?]

    2. CFO Fired [How fast can you forge his signature all those PO's you've been wanting.]

    3. Stock Devaluation [Speed selling]

    4. Stock Devaluation at a dot com [Resume update]

  • Get a DLT (digital linear tape) and create a cron script to run nightly/weekly according to your discretion. DLT's have dropped since there are other alternatives, so their inexpensive (considering you have a 15tb set up) or you could always have a Clariion purchased for this. Even a Netapp using bzip2 on a sys will do more justice than people realize. Many companies don't have any idea how important a backup plan is until the shit hits the fan and they're shit out of luck.

    Its always good to have at least 90 days worth of backups in case something may have been corrupted, whats more is the storage of that data. You don't want to just leave it lying around. Consider renting small storage space for a monthly dump of the tapes or keep em locked up in a secure place such as a safe, or encrypt the tapes so the average joe can't read it should they get their hands on them (the tapes)

    I hate even thinking about the idea of going to a new company, and having to deal with this since its a painstaking task, but once its done, its all a matter of following up on things, and making it part of daily/weekly/monthly work. It gets easier once you've done it and gotten it over with.

  • Yes, but for the record, what I protest is the chronic use of the word "interesting," not the posting of stories that are indeed interesting.

    ---

  • ...is identifying how we can pass the crisis-response knowledge on to the next generation of employees. One focus in US military exercises is not only on how each situation is dealt with, but how each person knows what to do in the first place. Rather than simply train each person from scratch then hoping they remember what to do under every single set of circumstances, the US Air Force often creates multi-purpose response checklists and "continuity" binders/folders that contain everything from basic response overviews to detailed information on how to deal with various problems. Saying "I'm not sure but the answer is written right *here*" is very nearly as good as having the answer memorized especially if the answer is available to everyone in the organization.

    It's a huge pain in the a$$ to prepare this material but it helps insure that there's always SOMEONE around who knows what to do and there is a source of info to check when the crap hits the fan. A little investment in time to create these documents pays off in a big way, one incident at a time. These are living documents instead of regulations, so they are continually subject to improvement or even disposal if they become obsolete or something better comes along.

    This kind of thing ought to work in the corporate world especially in a company where there is a measure of procedural inertia that carries on regardless of who is holding down each particular job.
  • Sounds like a lot of folks (at least in charge) with too little to do. The environment I'm in is constant crisis. I'd sign on for a 'NonCrisis Day' in a heartbeat.
  • I find that massaging the back right under the ribs also helps. Also, there is a program for the palm pilot [palmgear.com] that is helpful in keeping track when the next one will occur.
  • The classic reply:

    A lack of planning on your part does not constitute an emergency on my part.

  • I for one am tired of all these interesting stories. I demand boring stories on Slashdot!
    =\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\=\= \=\=\=\=\=\
  • Lordy God, you know people are getting jaded when they think Ethernet switches are a foregone conclusion. Knowing how to set up a network to reduce collisions, are knowing what a bus topology reeeeealy dates you these days.
  • Oooh good idea. I know what I'm coding up tomorrow.
  • Considering how many times I've seen our non-technical employees warn the company about hoaxes and spread email viruses I wouldn't be surprised if these people would give out dangerous information to a clever social engineer.

  • Actually, my company has a daily drill.
    Someone codes something good and shouts something like 1 4M S000 13373 D00DZ!!11 (or something else in 13373 speak), then we beat the ever living crap outta them.
    I'd like to see a script kiddie pop their head up in my company.

    BTW - This article sounds like a preface to a new BOFH.
  • Look at all those posts in the beginning marked as redundant.

    Those people are merely sharing how common crisis mode is.

    Even if it is just due to managers believing that you can do with half the staff, or half the hardware, or half the budget you need to get the results they want.

    Maybe it was a silly question to begin with, but it sure seems like whoever had the 150 moderator points had a problem with that much unexpected opinion.

    Check out the Vinny the Vampire [eplugz.com] comic strip

  • Well, crisis has a little different meaning in the Milittary. Try codeing in MOPP-4 (full chem) gear. Its not fun. Yes we have the regular bomb threat practice and the runs Tornado Shelter, both practice and real, I'm stationed At Tinker AFB in Oklahoma. Its a very, for lack of a better word, "Interesting" Job.

    Sanchi
  • We got a calendar that had things like "SecYOUrity" on it and "Suspect the unexpected! Delete unexpected email!"

    I was thinking though, technically most mail that I get falls under the category of "unexpected" :)

    dynamo

  • Well, I have to admit, it's solid in theory, but my experience ( both electronically and physically ) is that folks like to loose all their "cool" in situations of a critial nature ( SysAdmins not excluded ). People like to run during fire alarms ( when they're fot real ), tornados, when their the targets of crackers, etc. I guess drilling probably does give one person a very cushy job though.
  • For me, a good month is a month when I can get one week of project work done, where I'm not constantly being called away to put out fires.

    It's disappointing to see that as time goes by, more and more of my work putting out fires. It reflects badly on the management of the organization as a whole, but that's the nature of IT. You don't hear from people unless thay have a problem. Crisis week once in a while huh? Try having crisis week three times per month.


    --
  • Most likely running under Netware...
  • Well, I see two ways crisis week could work for me:

    "I wouldn't have that code finished until next week, I'm practicing having crisis this week."

    OR

    "NOooooooooo! this is the third time this week I've split bong water on my keyboard."


    --

  • Re:No need ... (Score:3)

    by Lemmy Caution ( 8378 ) on Tuesday May 15, 2001 @12:01PM (#221598) Homepage
    I know you're kidding, but I've recently rebelled against a perpetual sense of crisis at the workplace. It's the product of an over-caffienated Calvinism or something, but in many workplaces there's an unfocused and pervasive attitude of constant emergency. When any of it drifts my way, I now have a general response: if there's something actually and concretely urgent that is comparable with a child being trapped under a vehicle, then I'll freak out with the best of them. Otherwise, it's not really an emergency; rather, it's a frantic toadying, and I'll have none of it.
  • I have brought this up to my superiors several times and then get reminded that we only have 1 link to the "internet" and am told there is no need. They don't understand that attacks can come from within and that even 1 outside connection is all it takes.

    I've started putting some info together on possible situations and what should be done but but I'm not a security guru and could use some tips on possible senarios or some form of an example that might wake the management up in this company.

    And for those that seem to think you are reading between the lines, thats not an open invite to attack.

  • This previous "Ask Slashdot" [slashdot.org] discusses this, and deserves reference on the current subject.
    Prevention of social engineering is critical to corperate and personal information security.

  • 'Crisis Week' ?? (Score:3)

    by SpanishInquisition ( 127269 ) on Tuesday May 15, 2001 @11:46AM (#221601) Homepage Journal
    Is that between 'Major Layoff Week' and 'Sexual Harassement Week' or is just after 'Clueless Management Week'??
    --

  • Uh-oh.. (Score:3)

    by OblongPlatypus ( 233746 ) on Tuesday May 15, 2001 @11:52AM (#221602)

    You can just imagine the script kiddies already getting busy, tracking down this company and taking advantage of the situation..

    ".. and now you can observe how easily the intruder is gaining access to our classified documents. But don't worry folks, remember; it's just a simulation!"

  • Army Installations (Score:3)

    by Calle Ballz ( 238584 ) on Tuesday May 15, 2001 @12:25PM (#221603) Homepage
    I work on an army base. About a month ago, the entire Installation had our little practice excercises. It lasted one week, and we made it all the way up to Threatcon DELTA (The highest level of threat). During this time there was...

    Bombings

    Spies Peering into places

    Security Checkpoints almost everywhere

    Attacks on networks (software AND hardware)

    Power outages

    Phone outages

    All simulated of course. From the Army's standpoint, it was a great training excercise. From the contractor's standpoint, it was a huge hassle, but I learned a lot regardless. That was my experience though.
  • And it's practiced every month or so. Luckily, I can usually keep track and bring flowers to lighten the mood. I find the best thing to do during crisis week is to just shut up and get out of the way. Let the experts handle most of the issues.

    Dancin Santa
  • I accidentally opened the email. I didn't think IT would send me a virus.

  • VBS anyone? (Score:3)

    by FFON ( 266696 ) on Tuesday May 15, 2001 @12:13PM (#221606) Homepage
    at the failed dot-com i worked for, our employees were the double clicking, attachment getting, outlook using, microsoft sheep that graze the non-technical side the the IT industry. And as training, i as sysadmin would send bogus emails with VBS attachments that just open up a browser and took them to a page on our intranet that said: PLEASE DON'T OPEN VBS OR EXE FILES FROM YOUR EMAIL... then script would email me and i'd go have a chat with the person.... This worked wonderfully! People hate to be humilated...

  • Yes they do! (Score:3)

    by cavemanf16 ( 303184 ) on Tuesday May 15, 2001 @11:43AM (#221607) Homepage Journal
    For the last two years (at least, I've only been here 3), my company I work for has done presentations on security risks to our IT infrastructure. These presentations have been more of a "How do you prevent X from happening.", but do cater to informing the non-IT workers how to protect our data, and their own. I think they have been good wake up calls, even to me, and I work in an IT environment of the company. But certain issues that the regular Joe brings up in the meetings do get addressed and are taken very seriously by our Security folks. If your company isn't having meetings regarding IT security these days, you're missing the boat!

  • Drill: delete that e-mail! (Score:4)

    by Telcontar ( 819 ) on Tuesday May 15, 2001 @11:44AM (#221608) Homepage
    Attention! E-Mail coming in! It has an evil attachment; keep your eyes away from it! Don't open it, lest civilization as we know it ends forever!
    Now try to find the delete key... press it... done. Ah, life can go on now.

  • Re:Every week is a crisis week :) (Score:4)

    by selectspec ( 74651 ) on Tuesday May 15, 2001 @12:13PM (#221609)
    Yeah, software companies should have "Stand Down" week, where the company goes off crisis mode for 7 days and people go home to sleep.

  • Disaster Recovery facilities (Score:4)

    by AJGriff ( 94198 ) on Tuesday May 15, 2001 @12:32PM (#221610)
    Our company tests all of our major systems once a year with a company called Comdisco [comdisco.com]. They provide us with hardware (servers, disk arrays, tape drives, everything) that we can then use to simulate and test our disaster recovery procedures. They are a very large company (the largest in their industry I believe), and they can provide you with any type of hardware you need; everything from large mainframes down to small Intel based servers.

    Along with allowing us to test our procedures once a year, in case of a real emergency, our critical systems can be reloaded at their facilities and brought back up until the neccesery repairs/reloads are performed here.

    When you're doing a test, or an actual disaster recovery, they also have a full staff of experienced sysadmins to help. This is really valuable as even the most experienced sysadmin doens't get as much practice at disaster recovery as those guys do.

    As you can imagine, services like this aren't cheap, but they are aimed at large companies that need this type of protection. They also have a whole host of other services they provide, mostly network monitoring services; this may include testing your site's ability to defend against a hacker attack, but I'm not sure. If you work for a large (or growing) company and don't yet have a company providing these services for you, I highly, highly recommend Comdisco.


  • Ensure the bullet misses the foot... (Score:4)

    by Ian Peon ( 232360 ) <ian&epperson,com> on Tuesday May 15, 2001 @12:20PM (#221611)
    I spent a few years in the Navy living on-board a ship. We had fire drills almost daily at about 6 pm.

    Then one day, we actaully had a fire, at about 6 pm. Three of us were containing it, and called the QuarterDeck (front office) to sound the alarm, which they did, except they announced that it was a drill!

    Sooo, the people who usually run the drill (officers=managers) called the QuarterDeck and told them to cancel it because there was no drill for the day.

    Needless to say, we spent a VERY LONG time on the phone before the QuarterDeck got the story right, and the fire crew finally arrived.

    Moral of the story: Don't get so caught up in doing drills that you miss the actual fire!

  • Intrusion Detection (Score:4)

    by iomud ( 241310 ) on Tuesday May 15, 2001 @11:51AM (#221612) Homepage Journal
    Marcus Ranum [ranum.com] gave an interesting talk on intrusion detection systems and security including physical threats at ALS last year. I'd also recommend secrets and lies [counterpane.com] by Schneier. It also takes an interesting look at physical security issues. As for crisis week the last one I can think of was Y2K but that wasn't really a mock up type thing. The only other crisis preparedness we were trained for was 'fire'.

  • My company did this (Score:5)

    by jfunk ( 33224 ) <jfunk@roadrunner.nf.net> on Tuesday May 15, 2001 @12:52PM (#221613) Homepage
    Of course telling people that you're going to have a simulated crisis is not very effective at all. It just has to happen without warning or the workers are definitely going to be prepared.

    One day I came in to work and I was told that the CVS server went down. The support staff knew exactly when it went down because NetSaint sent messages to their phones.

    I'm not normally support/admin, but I have experience in it so I jumped in to help. Here is what we did:

    - Went to the console and tried to boot it up. No go

    - I booted from a rescue disk and tried to boot it that way. Nope

    - Tried to mount the partitions, found that the partition table was gone

    - We then split into two different efforts: I mentioned gpart (guesses lost partition tables) and started running it with various options while the other team began rebuilding the server from backups

    - gpart didn't work so I just partitioned it again with the original settings (I've done that successfully before on a home computer)

    - That didn't work, but the replacement server was ready by then so we plugged it into the network

    Once the backup server was up the head of development announced that he had replaced the CVS server's hard drive with a blank one early that morning.

    We all wrote reports on what we did and, while we were pissed for a minute ("You WHAT!?!?!?"), the drill was determined to be a success.

    I was freaked out mainly due to the fact that I volunteered to help out... Me and my big mouth :-)*

  • Oh, don't even start... (Score:5)

    by devphil ( 51341 ) on Tuesday May 15, 2001 @12:02PM (#221614) Homepage

    I sysadmin for a government research lab. You'd better believe every week is an IT crisis week. If it's not crackers in China looking for revenge for the embassy accident, it's some dumbfsck college kid trying to telnet past the routers or something.

    Those aren't the crises, though (the routers keep those jerks out). The actual crises begin when the logfiles get too big to fit on the backup tape. Then I have to scrounge around to find more tapes, 'cause they won't let me buy any more on the government budget (yes dammit I'd raise my own grandmother's taxes if it means I have money to buy backup tapes), and then I have to decide whether the stuff currently on the tapes can be sacrificed for the holy cause (backups! backups always take priority!). This decision-making process usually requires some caffeine, and the single soda machine within reach charges a freaking dollar for a 20-oz bottle, so there's another twelve or thirteen dollars gone.

    Don't talk to me about "planned" crisis week.

  • Citing CISSP (Score:5)

    by joq ( 63625 ) on Tuesday May 15, 2001 @11:51AM (#221615) Homepage Journal
    This is one of the topics covered in the CISSP exam [isc2.org], I think the CISA also has it. Methods for disaster recovery, which are often ignored by many companies. Often I wonder how much a company has prepared for a disaster, via way of anything imaginable, hurricanes, fires, break-ins, etc.

    Personally I think companies grow too fast and focus on growing, growing, growing, rarely stopping to take the time to implement measures against disaster recovery.

    One of the things we do @ my place is once every other month we have a sit in with beers, pizza, etc., and focus on security via way of games. Why do you need a safe password is based on a guess your co-workers info to see how much we can gather by knowing them to see if we could guess their pw's, we also have a twist on Jeopardy where we use the names obtained from Attrition.org, and make a question about the company, so we could say "yes this company was owned this/last month" in order to make our workers aware of the risks involved on the `net'.

    Its better than ramming security down their throats and constantly lecturing people. We also have little twists on dealing with all sorts of issues, voicemail management to avoid having pw's cracked, social engineering games, and makeshift scenarios where someone comes in to social engineer their way into information.

    keep us on our toes ;) ... For those with higher ranking positions I suggest you go out and get the "Information Management Handbook -- Tipton/Krauss" which has tons of informative information regarding safeguarding data, disaster recovery techniques, etc. Its one of the best books I ever bought.


  • OT: Interesting (Score:5)

    by rkent ( 73434 ) <rkent@post.ha r v a r d . edu> on Tuesday May 15, 2001 @12:14PM (#221616)
    I don't know why it just occurred to me with this article, but has anyone else noticed that the editors described damn near everything as "interesting?" A quick search [slashdot.org] reveals 32 occurrences in May so far. If you include April, the number rises to 101. That comes to a little over 2 uses per day, and considering there are only several 2-3 sentence articles posted daily, that's a pretty high "interesting" density.

    I'm glad the editors are posting stuff that piques their interest, but maybe it's time for a bit more editorial creativity? A vocab building class perhaps? Or maybe they should change the site name to "Slashdot: An interesting idea."

    [ yes, this is offtopic. It's probably also flaimbait. But I, for one, think it's funny. Or at least intersting. hehehe ]

    ---

  • Ah! Another NT domain...

  • Comment removed (Score:5)

    by account_deleted ( 4530225 ) on Tuesday May 15, 2001 @12:46PM (#221618)
    Comment removed based on user account deletion
  • We don't bother with simulating an IT crisis, we simply allow people to log into the network and do their daily tasks.

  • Caffeine Crisis (Score:5)

    by suss ( 158993 ) on Tuesday May 15, 2001 @12:16PM (#221620)
    Beyond the normal fire drills or chemical spills, a new addition was 'Attack on IT Infrastructure'.

    They took away the coffeemaker?

    must... have... caffeine... to... code...

Slashdot Top Deals

He has not acquired a fortune; the fortune has acquired him. -- Bion

Close