Network Penetration Scans and Executive Reaction? 434
LazloToth asks: "I'm sure some of you have had this happen: your company pays the big bucks for a 3rd-party security audit and, when it comes back, you get called on the carpet for all the supposed 'holes' in your network. When you see the report, you recognize that it comes from a well-known open-source security scanner, and that the 'holes' in question are so obscure as to be meaningless. I told our risk management VP that to fix every item cited - - many of which were false positives or completely out of context - - would be next to impossible for our small IT staff, and that some of the fixes, if implemented, might have deleterious effects on an otherwise smoothly running operation. How do you handle these 3rd-party security people who make mountains out of every molehill?"
Here's how I would handle it. (Score:5, Interesting)
Sit down with your boss and explain what each open port is and why it is open. Then explain what happens if you close that port.
Lock everything down tighter than fort knox, starting with your bosses machine (Yes sir, Im sorry you can't surf the internet, we closed that outgoing port because it was a security risk)
One of these should work (or get you fired) either way, you don't have to deal with employees upset because their VPN or Remote Access doesn't work.
Fix all the holes (Score:1, Interesting)
The weakest link... (Score:5, Interesting)
Every chain is only as strong as its weakest link.
This holds true in the military area, more than everywhere else. I work in environments that are very sensitive to security, and we take such external reviews extremely seriously. There's no such thing as an "obscure" or "irrelevant" weakness.
Unlike most vanilla companies, we can't afford to let things slide, security-wise. Knowing that your clients are prime target for highly professional black hats and (not only industrial) spies is highly motivating. This includes (of course) penetration testing (conducted both internally and by independant contractors), but also exclusive use of open source code and internal code auditing. As an aside: personnel (HR) auditing is also very important, if not even more so than technical aspects!
Sure, most companies don't need this level of security awareness and can get away with being "pragmatic", but don't complain when your client database (with all the goodies like credit card data etc.) gets compromized!
it's haaaard work (Score:5, Interesting)
Since you don't cite any examples of these issues, I would bet you're one of these people who think running PHP with register_globals on is a "molehill?"
Cite some examples, or else this looks like you're complaining that tightening security holes would be
Re:This is why I love my job (Score:1, Interesting)
Having a third eye doesn't hurt as long as you are confident in your abilities and stand behind your work.
Sort of like a lawyer does, never asks a question that they don't know the answer to. A true IT professional would never do an audit they don't know the outcome of.
Shoot, I can't believe I'm give this advice away for FREE! now pay me money!
Comment removed (Score:5, Interesting)
How to handle Security issues? (Score:2, Interesting)
How to handle the security report? With the same seriousness as your boss, he signes your paychecks after all.
Re:Its their job (Score:3, Interesting)
Yes...and this is why they should be providing context whenever possible to the "holes" they find, and verifying false positives (or qualifying them). I work for a security company, and we're very careful about this. For example, on many systems when a daemon is patched, the banners are not updated and so we'll see fully patched servers that flag on having vulnerable versions of software. We've seen this time and time again, and know that it could be the case each time we get that result. We either manually verify the finding in each case, or in our report, we explicitly state this for each such finding (if we can't verify due to the scope of work).
My advice to you is this: Stay ahead of the game. While it's not so easy to duplicate the work of qualified security assessors who will provide a quality and carefully-checked deliverable, it's pretty easy to do what these script monkeys did, and thus know in advance what they'll say so that you can respond back to management. And while you're at it, pointing out that five figures were paid for something you did in a spare maintenance window for the hell of it helps protect you as well.
Ultimately, it sounds like the vendor who did this assessment sucked, and it's just another case of "management paid to bring a crappy vendor in, and it made my life hard in _____ way." Fortunately, unlike most such situations, this is one where you can actually anticipate the way they'll screw up to some degree.
Re:Its their job (Score:5, Interesting)
Second best - sit them down and ask them to demonstrate the problem by breaking into your system NOW. Make sure it's a linux or bsd box, at a console, not a graphical login, and don't give them a user name or password. Most of these weenies are only comfortable with Windows.
Third best - tell them they were running nmap against your honeypot, not against your real network. They won't know if you're lying or not.
Re:Its their job (Score:3, Interesting)
Maybe I'm being naive here but I would hope that the "risk management VP" knows something about risk management. So, the approach I would take is to categorize the risks: seriousness of vulnerability, difficulty of fixing, priority.
If you break it into priorities, and put in some effort estimates, and the VP says, "fix all of them", that's tough for you. More likely he will stop somewhere in the middle and draw a line. Then everyone wins: auditors have been picky, you have made a technical risk assessment, boss has made a decision.
Re:Its their job (Score:2, Interesting)
Re:Its their job (Score:2, Interesting)
What I would have preferred would have been to have the people doing the testing verify a whole/vulnerability before it got bounced to me. This, IMO, is what they should be selling, not the raw output of some scanner, but the service of running the scan and then verifying/interpreting the results. (Of course this takes actually having a clue as to the relvance of the results.)
Re:Its their job (Score:1, Interesting)
I'm a mechanical engineer for a small manufacturer, so I'm not supposed to be doing IT. However, I have a good programming background and have set up and run a number of servers, so I know a thing or two. Everyone in the company knows that I am the most knowledgeable person there when it comes to computers, except, seemingly, the owners. The official sysadmin doesn't know much. The majority of what he does is to hire outside consultants, many of who do a really shitty job. We've been to data recovery twice. We've lost data. We have all kinds of chronic problems.
I never even get consulted on company IT. Often I just fix things while they are pricing out solutions. It's really annoying having to force your help on someone who doesn't want it simply because it's the only way you are going to be able to do your job. It's even more annoying to watch someone getting paid consultant rates to do a poor job when I could do a better job for free on the weekend. If only the job market was a little stronger.
Obscure? (Score:3, Interesting)
Make an ass out of them in front of your Execs.. (Score:3, Interesting)
Re:Get a new consultant (Score:5, Interesting)
The poster obviously is not in the position to `get a new consultant'. His problem is how he can hit his management with the clue stick.
Let me tell you a story that happened just a few weeks ago: I'm the CEO of a consulting company that does quite some security work. We were brought into the following situation: A customer of an outsourcer got an `independent' security audit by HP. The HP folks took the (actually very good) CIS benchmarks and demanded that each and every item of that benchmark is followed to the letter. As part of that, they demanded that the NFS and Samba servers are turned off.
There's just one small problem -- the actual service the outsourcer was providing to the customer is -- tada! -- file service over NFS and CIFS! The outsourcer pointed this out to their customer's management. That management is a bunch of morons and just told them back: But this is a security audit of HP, they know their thing! So they had to bring us in, to give their opinion `management cloud' by creating pretty PPTs.
Even though we earned quite some money on that job; I would have prefered to work on really improving the security, in particular the processes, instead of fencing unprofessional HP security `consultants' and idiotic management PHBs.
Re:it's haaaard work (Score:5, Interesting)
The poster had stated that the report came from "well-known open-source security scanner" which I can only assume means that it was generated from Nessus. As someone who runs Nessus on a regular basis for my company I have to say that the reports generated from nessus can be next to useless if not properly interpretted.
For example it will flag our RHEL boxes for running Apache 2.0.46 due to some obscure DoS or bug. Recommendation: Upgrade to latest. However it doesn't take into account that Red Hat has backported the fix into 2.0.46 and that RH Apache 2.0.46 is not vulnerable.
In addition, Nessus bitches about everything it sees, such as mail.domain.com is listening in on port 25. This is not a security risk, but rather intended behaviour.
I found myself in a similar position last year when a user brought in his home laptop and scanned the internal net with Nessus. This user brought the results to upper management at my company without even talking to us sysadmin folks. The manager freaked when she saw her servers so "vulnerable" and asked the sysadmin manager "what the hell is going on?".
Fortunately I had been conducting weekly Nessus scans myself. I showed my manager our archive dating back for months, and explained how this is prone to false positives. Explained how we had taken care of the real problems, and what can show as a false positive. He was impressed, went back to the other manager and explained the rest. In addition he had the user suspended for a week without pay for violating the terms of service for our network.
Long story short, cover your ass and run your own scans. Take care of issues as they come up. If a consulting company comes in and just runs a Nessus scan on your network, explain to your managers how the company is not offering anything new and how they haven't put any effort into interpretting the results.
It's not about spin, it's about interpretting what a security risk truly is.
playing the victim (Score:2, Interesting)
So why was the audit asked for in the first place and why did you not have at least a modicum of management control over the process? You should have gone in, hand in hand with management and looked at the result in unison, not being subjected to it - in the spirit of learning, not generating fault. Clearly, this audit was set up to generate fault, whether through management caprice or someone reading that it was a trendy thing to do.
My opinion is that you screwed up by permitting yourself to prostrate yourself to this white-hat audit without being part of the process and making yourself a beneficial part of the results; not a victim.
Not in the notion of the "not my fault" notion of management, but in terms of engaging the organization in demanding beneficial analysis and results, and working with them to improve your processes.
Being dive-bombed by a 3rd party means your management has a poor view of your organization or at least, you are communicating poorly with them.
Stop being a victim. Get your ass in gear.
Re:Its their job (Score:3, Interesting)
I set up monitoring on the network so that if anyone started to do anything funky on the network my terminals would beep.
I would then printout a piece of clip art with hand cuffs on it.
Trace down the ip address. Then walk to the correct office an say "Hi, are you doing something strange on the network?"
When they said the were, I would hand them the paper with cuffs and ask them to let me know when they were done.
After a couple of years, they started calling me in advance so I was not bothered.
Re:Its their job (Score:3, Interesting)
So, one day I get a call that there's a serious problem with traffic coming from the server's IP, and if I don't come talk to IT's network guys tomorrow, they'll shut things down. Of course, they don't bother to tell me what the problem is in the e-mail; I think they were deliberately trying to sound vague and ominous.
Anyway, I go in, and they hand me this 40 page report that claims to show hundreds of security problems, mostly with software that isn't even installed on the server -- or can't even be run on OS X. They also claim the server seems to be infected with something, which would have been a neat trick, given the total lack of OS X viruses. The report was basically used as a prop by the IT guys to 'put me in my place'. They wouldn't let me leave with a copy either, presumably because they realized (once they figured out I knew what I was talking about; I think they had previously assumed that since I wasn't in the IT department, I must be clueless) I could go through point by point and knock everything down.
Anyway, I pretty much blew them off. I watched network traffic with snort for awhile to see if there was any kind of actual problem (portscans originating from my IPs, or something along those lines), and I never found anything but a couple of false positives. Eventually, I just adjusted the firewall settings a bit so the IT guys couldn't see what I was doing. (Isn't that was firewalls are for? Keeping idiots you don't trust out of your network?) That seemed to solve the problem. Could have been nasty if they'd actually tried to take that report to someone to 'prove' that I didn't know what I was doing, though. I'm not sure I could have explained the report's bogusness to someone without the right technical background.
It's called a 'Risk Analysis'. (Score:3, Interesting)
Your basic risk analysis takes a look at all of the vulnerabilities on the system. For each one, you list the following:
- the likelihood of that vulnerability being realized
- the impact if that vulnerability were realized
- any mitigation that has been done to reduce the chance of it being fully realized and exploited.
Of course, management likes numbers, so you rank each item from 1 to 10 (or 1 to 100, or whatever), using whatever scale you want (so long as you're consistent in your rankings for all of the items). Then, you use the secret fomula : For the top 10 items (or however many you feel like, you come up with some rough estimates on how much it would cost to fix or reduce the impact, or otherwise mitigate each of the problems.Note: Some people will say that the 'impact' should be a dollar amount to signify the damages done to the company... but it's impossible. How much is a human life worth? Is it worth more than the company losing millions of dollars in sales? How does it compare to the loss of reputation if your clients found out about whatever it was?
Example: There is a real vulnerability that you may have an electrical fire. The threat of it happening however, tends to be very low, if the building inspectors did their job. The impact, if this happened on a weekend could result in the lost of the entire building. Countermeasures include fire extinguishers, sprinklers, temperature alarms, off site backups, redundant servers, etc. You can never get rid of the vulnerability, because there is always a chance of that fire happening.
Example 2: There is a possibiliy of all of the system administrators quitting, leaving you with no operations staff. This can be mitigated by treating them with respect, not forcing them to wear ties to work, and paying them better.
Use this to your advantage. Don't fight the report, done by someone who knows enough to schmooze the boss, and get paid many thousands of dollars to click a 'run' button. Use it to get rid of those nagging little things that have been bothering you, that you've never been given a chance to sit down and fix.
Cowboys (Score:3, Interesting)
. . . then they are quite similar to most of the fly-by-night security companies in existance today.
They really are a plague. Typically a small number of university students, or recent graduates, trying their hand at "start-up dotcom". There are two or three guys who know linux, a little about cisco routers, maybe had a course where they learned about Nessus. There will be fast talking marketing and sales slime involved as well. They are all very young and inexperienced, none of them will have spent any time in a large company with a complex IT infrastructure. Their M.O. will be to approach a company with the output of a Nessus scan of the firewall and web servers, showing a whole bunch of false problems, and try to get a security audit contract out of it.
if you're looking for someone to do a security assessment or pen testing
These external audit companies don't sit around waiting for an IT group to give them a call, because they'd never get one. They will not approach the head of IT, but a sales or a CEO level person with nary a clue. They leverage their way in from the initial external scan of the firewall and web servers. They get permission to run an internal scan, then hand over an unedited Nessus report, hundreds of pages long with their invoice.
The term over here is Cowboys. They ride into town unannounced, pretend to save the day, and ride into the sunset after claiming their reward, never to be seen again. Their victims, of course, are the struggling IT departments like the OP, who have done what they can with their limited budget, and suddenly have to answer to a mostly worthless Nessus report.
the AC
Re:it's haaaard work (Score:3, Interesting)
Because he didn't scan any of the machines that I work on. We are an offsite Gubmint facility, with each project having their own administrators. I, myself, work on a project.
The other administrators did notice, but assumed it was my scan since it came from an internal IP. I did go over IT infrastructure policy where it states that all scans are come from itscan.domain!
Better yet, have security walk into his cube and escort him out as the scan is finishing.
Because he is an awesome developer and to lose him would set back a major project. Got to pick your battles. Besides I don't want to get a guy fired, unless it's blatant abuse. In this case the guy did think he was doing a service. And since the fiasco, he's been one of our greatest supporters. He understands our work is more involved than he had originally suspected.
I'll grant that intrusion detection is hard. (and you have to deal with false positives from your department) There are valid reasons not to do it. I just want to know if you have a valid reason for not noticing his scan in real time.
See above!
Most so-called "Net Security Consultants"... (Score:3, Interesting)
My employer recently went thru one of these and I prepared for it (I am the network admin) by writing a list of everything the consultants would find, and why they would find it and what could or could not be done about it short of completely unplugging the affected bunch of machines and users off the network entirely. I also wrote down exactly what they would find when they attempted a penetration test from the outside to try to come thru our firewalls. I sealed up all my reports into an envelope and got my boss and his bosses above him to agree to keep the envelope sealed and not read it until after the consultants submitted their findings report and they'd read it.
During the tests, the consultants could not break in of course, and I got accused of refusing to cooperate with them. I told them to their faces in front of my boss that they weren't even worth half their weight in dirt and were basically committing a con against us. (con + insult = consult).
After their report was finished and my bosses paid them and read it, followed by reading my sealed reports, my employer basically agreed with me they'd just wasted $15K and my network security talents have never come in question again. The consultants didn't even find everything that I already knew was wrong with our network, and I haven't been permitted to fix the stuff that really needs fixing because too many user will bitch about the inconvenience it would impose on them.
Security Scanners, Inc. (Score:3, Interesting)
One of our credit card processing companies got a wild hair up their ass about security. Security is a good thing, I fully believe in it. But they hired their own 3rd party company to scan us. Over, and over, and over again.
The 3rd party sent them a big list, where we were just on the friendly side of a passing score. I'm not pleased with "just" passing. They sent me the list, and "suggested" that we fix all these obvious holes in our security.
Some of them were that the sites resolved in DNS. Ummm, you go to example.com, it's gotta resolve.
Another was that we had a firewall up. Because packets disappeared into our network (dropped, instead of rejected), it was a clue to potential hackers that we had a firewall up.. {sigh} Ok, so our firewall did exactly what we wanted, and we get scored down??
The remainder of the list were assumptions. They (through fingerprinting) identified that we were using *nix machines, we are running Apache running on the web servers in question. At the time, Apache_SSL was about 2 subrevisions behind Apache itself, which made it impossible to stay with Apache_SSL, and pass their test. Their beef with it was that there was an exploit for Win32 and OS2 for the particular version we were running. I wrote them a nice email and said "Ok, so there's an exploit for Win32 and OS2 for that version, but we're running on *nix".
The temporary fix for the Apache "warning" was to not display the version of Apache. I later changed over to mod_ssl, and stuck with the current version.
We still get quarterly reports from them. I sigh every time I see them. They just piss me off. Not that we're getting a security review, but the fact that I have to explain why perfectly acceptable things are listed. I can never get my score to 0 threats. Even if I firewalled off the machine, so they couldn't see it, I'd still get points against me, because they can see there is a black hole, where they know there is a machine. {sigh}
I glance over the list when it comes in, and look for anything interesting. Do they have anything relevant to tell me? Nope? Ok, put it off til next week to decorate around their mental problems. Most days, I have real work to deal with, and don't feel like doing stupid tricks for their entertainment. Of course, if I have the time, I love messing with them. Let them wonder why I'm running Apache 4.9.1 on an unknown platform.
Re:Its their job (Score:3, Interesting)
Yeah, and that will make you look...co-operative, right?
I've done security consulting for years: tens of Nessus scans, web app tests, pen. tests etc. From this background I have some points here.
One clear problem for a third party consultant is that the risk level assignment is not necessarily as clear cut as the Nessus/ISS/whatever report says. We've never given a client a report directly from the tool, but have written our own detailing the problem and in what circumstances the problem is exploitable. This manually compiled report is definitely the killer when project price is concerned. Web-based scans with automatically generated reports are so much cheaper...
Moreover, we usually work WITH the sysadmins instead of against them. This is a key thing in a successfull security audit. Most sysadmins are not security experts and if they happen to be, they still do not usually have the time to do a thorough sweep of the whole network. The sysadmins in my experience have usually been very HAPPY with our results. In all company internal scans there have been major holes, but after our report, they know exactly where they should put the time/effort to enhance their security and what patches/fixes/tools to use for this.
Besides, in my experience, most of the time sysadmins have not been given any direction whatsoever on the desired security level of the systems. So in the absence of any direction, the audit can NOT claim lack of compliance. We can only say that because the mgmt hasn't committed to security, their systems have ad-hoc security, i.e. security is occasionally good in spots where someone has had the time and clue.
Regards, a GSNA