Protecting Your Enterprise Network from Vendor App Servers? 258
anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?"
"Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.
Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'
What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"
Re:Layer 7 Firewall (Score:2, Interesting)
It's a pitty they don't teach this stuff in CS:
http://www.dyadsecurity.com/papers/rbac.html [dyadsecurity.com]
http://www.nsa.gov/selinux/papers/inevit-abs.cfm http://www.acm.org/classics/sep95/ rtsp://media-1.datamerica.com/blackhat/bh-usa-00/
Two things... (Score:3, Interesting)
If a vendor is unsympathetic to your concerns, it's up to you to find an alternative or work around them. As you explain, the second option is not always possible when they require access to a number of services at a fundamental level. The worst cases of this occur when you have one or two vendors to pick from for a given application. My suggestion is then to design the application yourself within your security parameters and functionality requirements -- as many people do not have that capability within their own ASP (otherwise they'd do it already) you might want to use something like Sourceforge and contract a team overseas to do it cheaply, supervising the project from here and optionally open-sourcing it after it's built. Then you've got something designed to your parameters without support or upgrade costs especially if the community digs what you've built.
Vendors are asshats (Score:4, Interesting)
In the long run though, I've figured that these lusers are going to be more trouble than they're worth, and am talking the boss into replacing the NetApps and Junipers (routers) with Gentoo Linux boxen. So in short, don't protect, just reject!
Re:Consultancy (Score:1, Interesting)
FYI: I worked with their Cisco Jockies and on a few occasions, their Server Admins.
Well hey, they are owned by Ross Perot.
Nuff said.
They can screw up without root (Score:5, Interesting)
Our antispam server is a dedicated server, but it still was screwed up once by the vendor. We were having a problem (only half our mail was actually being processed by the filters), and management directed that I go ahead and give the vendor support person access to the server (as the user the software runs under). He logged in to look at it, and within minutes the system was broke and mail was queueing up because the antispam software shut down and wouldn't restart. He had seen something unrelated to our problem that he thought didn't look right and decided to "fix" it for us. As soon as I got him to change it back, I told him to log out and removed his access (and they won't ever get it again).
After that, the only other time we considered allowing a vendor access was on a problem case that was over a year old that we had to have fixed ASAP. However, in that case, we were NOT going to allow remote access; the vendor was going to have to send somebody to us and we'd sit him down in front of the console (which would be logging) with us looking over his shoulder.
The only people that know your systems and network are your people (and you'll make enough mistakes). Vendors should not have access to change things at any point; at most they should get a "view" type account (they can look all they want but they can't touch). If something needs changing, they need to tell you and you can make the change (after evaluating it and having a back-out plan). For complex systems, you never end up installing software exactly the way the vendor specified; they are not going to know or understand changes you made for your local configuration (and how such changes affect other services and systems). Even a well-meaning "fix" can cause serious problems, and since it'll be your job to fix them, you better know what was done.
Re:Switch vendors (Score:5, Interesting)
I've worked escalation support for a large storage vendor for a number of years, and while I completely understand (and agree with!) limiting vendor access, I find that IT departments tend to be *very* unresponsive when the vendor support folks need their help.
It's not exactly efficient, for example, when I have to wait 7 hours for logs to be uploaded, 5 hours for the "network guy" to respond to a page and fix the duplex mismatch he created that's causing 50% of packets from our systems to clients to be dropped, weeks for the system admin to stop piddling around and get Sun on the phone when we have valid interoperability issues (Sun won't talk to most other vendors except through the common customer), etc.
In the meantime, in my experience, the CIO tends to lambast the vendors for "poor responsiveness" and "terrible support" - though the smart ones eventually realize that the IT staff is shooting themselves in the foot, and do something about it.
IT departments that are responsive to vendors, on the other hand, are a breeze to work with, and issues are typically resolved many times faster.
Let them run under VMS (Score:2, Interesting)
If you need to run more safely in x86, I would suggest having a look at SELinux (from NSA) which gives some better controls than usual. Pay special attention to protections of devices, especially some of the more dangerous ones like
Non-technical reasons! (Score:3, Interesting)
There is big marketing battle just behind you!
As a vendor, customers are idiots (Score:3, Interesting)
Well, try explaining to IT (not a smart bunch, after all they dropped out of CS (or never even tried)) that their DB du jour is not up to the task of our RDBMS. True 90% SQL compliance is a good start, but there it is impossible to create a full-featured app and still be completely agnostic.
Examples: MySQL is the worst. The last insert ID is a horrible concept that is not portible. Try finding it in Informix, it is impossible to find, but exists. Thent here is the MySQL timestamp 'feature' - the first timestanp field (and only the first) when UPDATEd is updated to NOW() regardless of whether or not you include it in the updated feilds.
The only reason why they want the DB changed is because they have lazy IT departments that want to do nothing. Their IT staff does not understand the complexities of SQL DBs to them it is just an DB, and "thay're all compabible through SQL" yeah, right. "What about ODBC?" ODBC is fine for basic record operations, but the moment you try to do anyhing advanced, you're out of luck.
I used MySQL as an example, but I've worked with several other Databases, and there is no Holy Grail of DB abstraction. There's much more to databases than inserting updating, deleting and selecting rows, depsite what IT thinks.
Re:Switch vendors (Score:4, Interesting)
That where the vendor's software is on a "black-box" machine that only they have administrative access to, and which runs no other software.
I'm (senior deployment engineer at) one of those vendors. Not only is our app fairly complex to administer, but every server that goes out contains a copy of the "secret sauce" -- not our application itself (which our bigger competitors could probably recreate in 9 months or less), but the data behind it (much more expensive and difficult to recreate). Consequently, our management is paranoid -- the servers are rigged to self-destruct (wipe the keys that allow them to decrypt the partition where all data is stored) in the event that any attempted tampering is detected, and can only be reenabled by the very small subset of our staff w/ access to the private keys.
Right now our app has components that run as 4 different users (to isolate breakins to the component where they occur), and includes a firewall and a VPN solution (both of which need root for obvious reasons).
Since it's our box that's running the app, and that same system does nothing else -- why restrict our access further? Absolutely, firewall us off from anything we don't need -- but restricting access to our box seems silly (and is something we'd consider only for a very large customer -- which is just as well, since the small folks we're selling to right now haven't had a problem).
In any event, I'm curious to hear how you'd respond.
An easy solution? (Score:3, Interesting)
Re:You don't (Score:5, Interesting)
If you have something go down on you which is costing several people's salaries per hour in lost revenue, you don't sit around for days trying to sus it out - you get the vendor to fix it. That's why you pay them for their support. You do pay for vendor support don't you?
They know their products a hell of a lot better than you ever will, and you'll likely have a contract saying it will be fixed in a stated timescale or they'll be paying you compensation. Often vendors will courier out replacement hardware instantly before the engineers have even sat down to log into the device, and those engineers will be of the highest quality if you've flagged a critical call. If you've triggered a bug in a piece of vendor kit, how long will it be before you've diagnosed it to be a bug? Then how long will it be before you get a fix? I know of at least one large company which has compiled and installed a new software image on a customer device within hours of a critical fault call being raised. Would they do that if you weren't paying for support? I think not.
For maintainence however, you learn how to do it yourself.
On a similar note, us logging RDP possible? (Score:4, Interesting)
Which makes me wonder why I can't record an RDP session somehow. This would serve two purposes: a) it would let me replay what a vendor had done later when it is more convenient and b) it would give me some proof if I later have to go blast a vendor for doing something absurdly stupid.
The same question for VNC or really any kind of remote management tool that is likely to be used on the Windows platform. Can any of them be logged and/or replayed?
- JoeShmoe
.
Re:Find a new vendor (Score:3, Interesting)
Dual auditing. My activity is logged by a system I have no control over, and that other admin is logged by my system. It's true there are ways I could cover my tracks but it would be apparent I had hidden something even if they couldn't tell what I hid. At which point my ass would no longer be covered (logging has advantages for both sides in that sense), and I'd be asking if you want fries with that.
That's how my shop does it at least.
Re:Tell them to screw off.. (Score:3, Interesting)
The manager needs to have the password in a safe someplace so that if you all quit at once he can get someone else to keep the systems going.
Otherwise, whats the problem with giving him root? Just because says tradition is that account 0 one unix systems is called root doesn't mean yours has to be called root. Rename the root account to something else, and create a no privliges account called root that you can give anyone the password to.
If nothing else that is a good way to weed out those who need higher access. The people who don't figure it out right away are either too incompitent to have account, or can be trusted to never even look at root until it is really needed. (which is left as an exercise to the user) Those who notice right away have enough of a clue that they might not break something.
Note that on a properly setup system you do not need a root password at all. sudo can run all the programs you need. At least that is what I'm told, I've never got my systems that far.