Protecting Your Enterprise Network from Vendor App Servers?

anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?"

"Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'

What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"

Layer 7 Firewall

[passthecrackpipe]

You need a Layer 7 Firewall, a.k.a. an application level firewall. Something like Zorp is a good start, but you probably need something with a bit more intelligence about the applications you are talking about though.

L7 Firewalls usually get a bad rap because they tend to be pretty fussy in setup, something you can't really avoind with this kind of stuff. Also, if I was in your shoes, I would learn to stop worrying and start loving tight-ass SLA's...

Tell them to screw off..

[Heem]

Seriously.

I built a network for vendor lab equipment that was it's own netowrk, connected to the main network via a file server with 2 nics and NO routing. Therefore, file could be transfered from the lab network to the file server, and then from there to any of the main networks, of course after they had been scanned for viruses and such, since anti virus software usually would not run on their equipment. My rules were firm and backed by higher-ups - if you couldnt get your equipment to work in the enivroment given, or with only minimal flexibility from me (for example i'd write scripts to move their files automatically to the main, backed up network or something simple like that) - then we will find another vendor to provide us a solution that fits. Period. I never had any problem with a vendor once they heard the terms. They won't make money off us if they can't make the system work in our environment.

Switch vendors

[31415926535897]

Our vendors are notorious for demanding superuser access to the boxes that support their applications.

There are a few things you can do in this case:
1. Get a different vendor (if the application truly does require su for vendor support).
2. Reqest different support staff from the vendor (do this if the app doesn't require su but the support staff is too lazy).
3. Learn how to support the app in-house

I am very serious on these points. You do not want to give root access to people that should not need it. If they say they need it, they have an awful application.

The place I work has a vendor area fenced off from the rest of the server room, and the vendors only have access to what they need. If they need more privileges, the IT guys watch them like a hawk until they're done.

make sure everyone understands what the problem is

[discogravy]

if the app NEEDS to run, you put it on a DMZ and let the world have at it. If they want internal access....make an effort to secure it and when they say they can't do that, get it in writing -- email will do if you've already got that -- and make sure you've secured everything you can. Not much else you CAN do, unless you're the boss.

Don't let them

[illumin8]

Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

Simple answer: Don't let them. This is standard operating procedure in the financial services industry. Do you think that a bank would EVER let a non-employee or non-contractor access any bank system whatsoever? Especially remotely? If these companies want to do business with you, they WILL play by your rules or you'll pick one of their competitors products. In my experience, the only companies that required remote access to their systems were ones that A. Didn't have a fully working product, and B. Had to have the developer log in constantly to patch binaries with the latest bug fixes just to get a semi-working product. These are not the type of companies you want to do business with in the first place.

Speaking as a sysadmin for a smallish financial services company that processes around 1.2 million transactions a month, I would NEVER allow any vendor remote access to our network. It just wouldn't happen. Even if I did want to give them access, there are rules and regulations that strictly forbid my giving them access. If they give you a hard time, make something up about a security audit or something.

Seriously, you are asking for trouble if you let them have access. Who's going to take the blame when one of their developers logs in and wipes out all of your company's data? Chances are they'll blame it on you and you'll be in trouble for their mistake.

VPN

[Julian Morrison]

Create a dedicated VPN. The misbehaving app server sees only this VPN. Everything it needs to access has a presence thereon, carefully firewalled to allow the relevant ports to open. Everything it doesn't need to access, is not even on the network.

service accounts...

[Anonymous Coward]

I'm an IT guy who does applications managment for an Air Force network and this is a problem that we have run into as well.

Exchange Service accounts, NetIQ accounts running as Domain Admins etc.

This is generally not a good idea, but for your average IT admin superuser domain level rights were the only way to have those programs function properly for the longest time. However, one simply solution that I have just recently tested with our NetIQ application managment Server is to create the local service account on say...our SMS server not as a domain admin, but just as a local admin. If I do this I can avoid giving blanket access to the entire domain to that program, and I can control which servers have that particular service account on them.

Obviously its not a total fix, although luckily for us, server 2003 has eliminated a lot of the most common "null session" issues that were such a pain to hunt down and change manually.

The only other thing I could suggest is creating a test-bed network and installing one of the suspect application servers at a time and using a sniffer capture program to see which ports have what sort of protocol activity over them. Document the information, test several application servers at a time, shut down uneeded ports and then implement it on your network. *with approval of course ;)*

I do hope that I haven't completely misunderstood your problem.

VPNS are handy...

[eldub1999]

We force the vendors to enter via VPN. we use the VPN gateway to restrict each vendor account's access to only the IP addresses of the systems they need access to. Further, we occasionally use a packet sniffer to watch certain vendors.

We disable the account by default and require them to contact us and tell us what they are doing (change control) before letting them in.

Works for us.

Re:Switch vendors

[Eudaemonic Pie]

Exactly right. Vendors never *need* root on our box. They often *want* it because it makes their job and their life easier. With properly applied permissions, there is very little a vendor cannot do just using the application owner id. The exception being if their app server binds to ports 1024 and they need a restart. Anything else, like oh adjusting permissions of files they don't own, applying OS patches, rebooting the box, killing processes they don't own, etc, etc aren't things I want my vendors doing *anyway*. Depending on the size of your shop, you may have controls or processes in place that require approvals before any of that works gets done, so why let the vendor go around your process if you can't?

Re:Don't let them

[RobK]

I agree with these guys.

1. Because you're smart enough to ask this question, you've shown that you should be capable of handling the boxes and applications (even if you don't have the time to do it)

2. The vendor should know their product well enough to give instructions or ask for debug information from you about the configuration/logs and so on.

3. Do you have the time to fix it when they screw it up?

I go through this all of the time

[flying_monkies]

From the vender side.

First: It's amazing how many people haven't got a CLUE. "Don't give it to them or get another vender" isn't an option. When you're running a 24x7x365 system and expect sub 1 hour response time from your venders (our customers do) you're going to have to give some.

All of the boxes we hook up have phone lines into them. If security is an issue on dial-in access, we set it up as follows:

Modem is enabled dial-out only, no dial in access.
If the box dials home, we contact the administrator, identify who we are and which box has troubles and have them enable access to the server/hardware and reset the root password temporarily on the box. This occurs only if it's something we haven't already configured to work with sudo.
We then keep logs of the entire session, then email it to the customer when we're done.
When we're done, we have them reset the root password and disable dial-in on the modem again.
If we require access to a network resource from the machine, we're onsite with the customer shoulder surfing.

We do this at secure military sites, financial institutions, etc. and have yet to have an issue in 20 years.

Re:Switch vendors

[Eneff]

Sure... I guess you can install ArcIMS yourself, but if my app uses ArcIMS, and I'm supposed to be installing it on Solaris, I need root access for the install.

At least, it used to... Luckily I haven't been the poor sod to do installs lately. :D

(I do remember the first time I had to install ArcIMS... what a piece of hell.)

Check-out the FreeBSD jail facility

[Diomidis Spinellis]

[I know this will cost me karma points.]

The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants. The jail facility is based on the chroot(2) implementation, but prevents well-documented means to escape chroot confinement, offering partitioning of the file system, process, and networking namespaces. The facility removes all super-user privileges that would affect objects not entirely inside the jail.

For more information read:

• Poul-Henning Kamp and Robert Watson. Jails: Confining the omnipotent root. In Proceedings, SANE 2000 Conference. NLUUG, 2000.PDF [freebsd.dk], HTML [freebsd.org].
• Poul-Henning Kamp and Robert Watson. Building Systems to be Shared Securely ACM Queue vol. 2, no. 5 - July/August 2004 HTML [acmqueue.org]

You don't

[bahamat]

You do NOT let outside users have access to production systems as root (or as any other user). When they ask, you tell them to stop smoking crack and make them tell you how to fix it. When they ask again, instead of root, give them the middle finger, and them make them tell you how to fix it.

You need to learn to do things on your own. Calling up the vendor and handing them root to fix your problems merely makes you vendor dependant, and in my opinion any SysAdmin who does that should be fired. If I were a manager, why would I continue to employ SysAdmins who only call the vendor every time there is a problem? I would be thinking to myself that not only am I paying this bozo a huge sallary, but I'll be paying for the support contract forever as well, and not just for one system, but for however many app servers you've got. Why not just hire someone for minimum wage to dial the vendor when there's a problem than pay someone who is suposedly highly trained but can do nothing for themselves?

I'm not a manager, but I am the lead SysAdmin for an Internet services company. We have about 40 servers and about half as many networking devices (managed switches, firewalls, load balancers, etc). Whenever we get a new type of device we do end up calling the vendor quite a bit, but we always make sure they teach us the solution and we impliment it. Over six months or so as I learn all the ins and outs and bugs of the device we no longer need to contact the vendor. I have a team of three people, and if one of them gave a vendor the root or enable password on any of our devices I'd have them fired for network security breach, and if they weren't taking proper steps to learn a new device on their own I'd have them fired for incompetence and replaced. There are too many smart people out there to keep employed dumb ones, especially for the price tag SysAdmins deserve.

Re:Layer 7 Firewall

[Mattcelt]

Application proxy firewalls work just fine - Raptor makes an enterprise-class AP firewall that does a fantastic job. They require more resources to manage than simple stateful-inspection firewalls, but they are much more secure when managed properly.

As far as Mandatory Access Control (MAC) goes, it is even more difficult to manage than an AP firewall, and a terrible pain to implement - ever tried to make a MAC model work in an Active Directory environment? Not easy...

And even if you just choose to implement the RBAC part of a MAC model, how do you define roles? Unless you have a very stratified and well-defined role structure for the people who work in the enterprise, it is a daunting task to set up roles. In one place I worked, there were ~10,000 employees - and 4800+ job titles. Not exactly conducive to role assignment, to be sure.