Forgot your password?
typodupeerror
Software Businesses Security

Protecting Your Enterprise Network from Vendor App Servers? 258

Posted by Cliff
from the deny-the-errant-app-servers dept.
anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?"
"Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'

What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"
This discussion has been archived. No new comments can be posted.

Protecting Your Enterprise Network from Vendor App Servers?

Comments Filter:
  • Layer 7 Firewall (Score:5, Informative)

    by passthecrackpipe (598773) * <passthecrackpipe ... m ['l.c' in gap]> on Saturday November 27, 2004 @12:42PM (#10931298)
    You need a Layer 7 Firewall, a.k.a. an application level firewall. Something like Zorp is a good start, but you probably need something with a bit more intelligence about the applications you are talking about though.

    L7 Firewalls usually get a bad rap because they tend to be pretty fussy in setup, something you can't really avoind with this kind of stuff. Also, if I was in your shoes, I would learn to stop worrying and start loving tight-ass SLA's...
    • Re:Layer 7 Firewall (Score:2, Interesting)

      by jrl (4989)
      They also get a bad rap because they don't work :).

      It's a pitty they don't teach this stuff in CS:
      http://www.dyadsecurity.com/papers/rbac.html [dyadsecurity.com]
      http://www.nsa.gov/selinux/papers/inevit-abs.cfm http://www.acm.org/classics/sep95/ rtsp://media-1.datamerica.com/blackhat/bh-usa-00/v ideo/2000_Black_Hat_Vegas_VK3-Brian_Snow-We_Need_A ssurance-video.rm http://www.radium.ncsc.mil/tpep/library/rainbow/52 00.28-STD.pdf http://hissa.ncsl.nist.gov/rbac/paper/rbac1.html
      • Re:Layer 7 Firewall (Score:4, Informative)

        by Mattcelt (454751) on Saturday November 27, 2004 @02:52PM (#10932089)
        Application proxy firewalls work just fine - Raptor makes an enterprise-class AP firewall that does a fantastic job. They require more resources to manage than simple stateful-inspection firewalls, but they are much more secure when managed properly.

        As far as Mandatory Access Control (MAC) goes, it is even more difficult to manage than an AP firewall, and a terrible pain to implement - ever tried to make a MAC model work in an Active Directory environment? Not easy...

        And even if you just choose to implement the RBAC part of a MAC model, how do you define roles? Unless you have a very stratified and well-defined role structure for the people who work in the enterprise, it is a daunting task to set up roles. In one place I worked, there were ~10,000 employees - and 4800+ job titles. Not exactly conducive to role assignment, to be sure.
  • Consultancy (Score:5, Funny)

    by KontinMonet (737319) on Saturday November 27, 2004 @12:44PM (#10931310) Homepage Journal
    Well... don't get EDS to work on it [slashdot.org]!
  • by Heem (448667) on Saturday November 27, 2004 @12:44PM (#10931314) Homepage Journal
    Seriously.

    I built a network for vendor lab equipment that was it's own netowrk, connected to the main network via a file server with 2 nics and NO routing. Therefore, file could be transfered from the lab network to the file server, and then from there to any of the main networks, of course after they had been scanned for viruses and such, since anti virus software usually would not run on their equipment. My rules were firm and backed by higher-ups - if you couldnt get your equipment to work in the enivroment given, or with only minimal flexibility from me (for example i'd write scripts to move their files automatically to the main, backed up network or something simple like that) - then we will find another vendor to provide us a solution that fits. Period. I never had any problem with a vendor once they heard the terms. They won't make money off us if they can't make the system work in our environment.
    • I'm senior deployment engineer at one of those vendors (well, not one of your vendors, but similar kind of deal). One of the things I've put the most time into is building a (OpenVPN-based [sf.net]) administration infrastructure that'll work damn near anywhere. (If we need to, we can even tunnel over HTTP -- hasn't come to that yet, though).

      Our integration components are likewise designed to be flexible and nonintrusive -- as much code as possible on our server, as little as possible on the system we're integrating
      • Any vendor that came to me and said "we've taken measures to make sure that the normal use of our product does not interfere with the normal use of your network" wins the bid.
  • Switch vendors (Score:5, Informative)

    by 31415926535897 (702314) on Saturday November 27, 2004 @12:46PM (#10931327) Journal
    Our vendors are notorious for demanding superuser access to the boxes that support their applications.

    There are a few things you can do in this case:
    1. Get a different vendor (if the application truly does require su for vendor support).
    2. Reqest different support staff from the vendor (do this if the app doesn't require su but the support staff is too lazy).
    3. Learn how to support the app in-house

    I am very serious on these points. You do not want to give root access to people that should not need it. If they say they need it, they have an awful application.

    The place I work has a vendor area fenced off from the rest of the server room, and the vendors only have access to what they need. If they need more privileges, the IT guys watch them like a hawk until they're done.
    • Re:Switch vendors (Score:3, Informative)

      Exactly right. Vendors never *need* root on our box. They often *want* it because it makes their job and their life easier. With properly applied permissions, there is very little a vendor cannot do just using the application owner id. The exception being if their app server binds to ports 1024 and they need a restart. Anything else, like oh adjusting permissions of files they don't own, applying OS patches, rebooting the box, killing processes they don't own, etc, etc aren't things I want my vendors
      • Re:Switch vendors (Score:3, Informative)

        by Eneff (96967)
        Sure... I guess you can install ArcIMS yourself, but if my app uses ArcIMS, and I'm supposed to be installing it on Solaris, I need root access for the install.

        At least, it used to... Luckily I haven't been the poor sod to do installs lately. :D

        (I do remember the first time I had to install ArcIMS... what a piece of hell.)
      • Remember - making life easier for your vendor has a direct and positive effect on your bottom line - it makes their product cheaper.

        Sure, lock them out. You have a problem with their software, call their tech support, and when they ask for access, tell them to go fish... but then don't bitch about the callout charge to get an engineer out to your site.

        Your call.
    • Re:Switch vendors (Score:5, Interesting)

      by ischorr (657205) on Saturday November 27, 2004 @01:28PM (#10931557)
      "If they need more privileges, the IT guys watch them like a hawk until they're done." ...And hopefully your staff *is* there for them when they need other access, questions about other pieces of the environment, etc.

      I've worked escalation support for a large storage vendor for a number of years, and while I completely understand (and agree with!) limiting vendor access, I find that IT departments tend to be *very* unresponsive when the vendor support folks need their help.

      It's not exactly efficient, for example, when I have to wait 7 hours for logs to be uploaded, 5 hours for the "network guy" to respond to a page and fix the duplex mismatch he created that's causing 50% of packets from our systems to clients to be dropped, weeks for the system admin to stop piddling around and get Sun on the phone when we have valid interoperability issues (Sun won't talk to most other vendors except through the common customer), etc.

      In the meantime, in my experience, the CIO tends to lambast the vendors for "poor responsiveness" and "terrible support" - though the smart ones eventually realize that the IT staff is shooting themselves in the foot, and do something about it.

      IT departments that are responsive to vendors, on the other hand, are a breeze to work with, and issues are typically resolved many times faster.
    • Re:Switch vendors (Score:5, Insightful)

      by theonetruekeebler (60888) on Saturday November 27, 2004 @01:48PM (#10931681) Homepage Journal
      Sometimes switching vendors just isn't an option. There are lots and lots of niche markets where there's that one tool that everybody has to run. For example, at a motorcycle dealerships most parts fiche CDs will only work with a specific parts/sales software tool---which runs on NT and "needs" to run as Admin, even though it is only an elaborate piece of middleware connecting a database to a few desktops running the application. It is (as of mid-2003) also a slow, buggy piece of shit.

      When the only alternative to required software is working by hand (or a major reverse-engineering project), you just gotta suck it up and figure out how to protect the rest of your systems from their arrogance.

      Generally speaking, it's a clear sign of laziness or incompetence on the part of a half-assed programmer to think he needs root for everything. Hell, Oracle doesn't need root to run, and it's a mighty damned complicated RDBMS suite. If you're stuck with one of these vendors, I urge you to make it clear to them that you are using their software because you are stuck with it, that given the chance you will jump ship in a heartbeat, and that the reason you'll never buy any of their other products is that their claim to "need" root is a sign of either ineptitude or a cavalier attitude towards their customers.



      • Generally speaking, it's a clear sign of laziness or incompetence on the part of a half-assed programmer to think he needs root for everything.

        Amen. The OP said that his company had a large IT infrastructure which means they are worth the vendors business. I urge him to apply whatever pressure he can on the vendors to change their ways.

        That way, little places like where I currently work might get the same benefit!
    • Re:Switch vendors (Score:4, Interesting)

      by cduffy (652) <charles+slashdot@dyfis.net> on Saturday November 27, 2004 @02:44PM (#10932030)
      There's a different kind of situation:

      That where the vendor's software is on a "black-box" machine that only they have administrative access to, and which runs no other software.

      I'm (senior deployment engineer at) one of those vendors. Not only is our app fairly complex to administer, but every server that goes out contains a copy of the "secret sauce" -- not our application itself (which our bigger competitors could probably recreate in 9 months or less), but the data behind it (much more expensive and difficult to recreate). Consequently, our management is paranoid -- the servers are rigged to self-destruct (wipe the keys that allow them to decrypt the partition where all data is stored) in the event that any attempted tampering is detected, and can only be reenabled by the very small subset of our staff w/ access to the private keys.

      Right now our app has components that run as 4 different users (to isolate breakins to the component where they occur), and includes a firewall and a VPN solution (both of which need root for obvious reasons).

      Since it's our box that's running the app, and that same system does nothing else -- why restrict our access further? Absolutely, firewall us off from anything we don't need -- but restricting access to our box seems silly (and is something we'd consider only for a very large customer -- which is just as well, since the small folks we're selling to right now haven't had a problem).

      In any event, I'm curious to hear how you'd respond.
      • The problem, as far as I can tell, is that vendors want root to the boxes supporting theirs - the database servers, mailservers, and backup servers that are controlled by the customer. Obviously, it's not an issue if you have root on a box that you're renting to a customer; the customer's sysadmins should simply firewall you off like they would any other untrusted box, then pierce the firewall as necessary to let you access what you need.

        I just can't see where a black-box app vendor would ever need continu

  • by discogravy (455376) on Saturday November 27, 2004 @12:46PM (#10931329) Homepage
    if the app NEEDS to run, you put it on a DMZ and let the world have at it. If they want internal access....make an effort to secure it and when they say they can't do that, get it in writing -- email will do if you've already got that -- and make sure you've secured everything you can. Not much else you CAN do, unless you're the boss.
  • by Anonymous Coward
    Only give temporary access to allow agreed changes to take place. Superuser isn't required to diagnose problems, only to fix them. This also gives some stability as 3rd parties don't enter and alter things as they please.

    The disadvantage is that there is an operational overhead -- but then there should be because if its a pain to change things then less gets changed.
    • How can you say that superuser isn't required to diagnose problems?

      What if the problem is caused by bogosity in a config file that only root can read?

      What if the logs produced by the application are only readable by root (or by adm)?

      What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?

      • What if the problem is caused by bogosity in a config file that only root can read?

        Then it's written wrong. I want the config files to have permissions for a user named $vendor.

        What if the logs produced by the application are only readable by root (or by adm)?

        Then it's written wrong. I want the log files written to a location for a user named $vendor.

        What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?

        Then it's written wrong. I want the
      • How can you say that superuser isn't required to diagnose problems?

        What if the problem is caused by bogosity in a config file that only root can read?

        What if the logs produced by the application are only readable by root (or by adm)?

        What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?

        All of these can be handled with a proper sudo configuration. sudo rights to cat the log files and config files, you vendor does know which files he needs to

  • Don't let them (Score:5, Informative)

    by illumin8 (148082) on Saturday November 27, 2004 @12:48PM (#10931343) Journal
    Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

    Simple answer: Don't let them. This is standard operating procedure in the financial services industry. Do you think that a bank would EVER let a non-employee or non-contractor access any bank system whatsoever? Especially remotely? If these companies want to do business with you, they WILL play by your rules or you'll pick one of their competitors products. In my experience, the only companies that required remote access to their systems were ones that A. Didn't have a fully working product, and B. Had to have the developer log in constantly to patch binaries with the latest bug fixes just to get a semi-working product. These are not the type of companies you want to do business with in the first place.

    Speaking as a sysadmin for a smallish financial services company that processes around 1.2 million transactions a month, I would NEVER allow any vendor remote access to our network. It just wouldn't happen. Even if I did want to give them access, there are rules and regulations that strictly forbid my giving them access. If they give you a hard time, make something up about a security audit or something.

    Seriously, you are asking for trouble if you let them have access. Who's going to take the blame when one of their developers logs in and wipes out all of your company's data? Chances are they'll blame it on you and you'll be in trouble for their mistake.
    • Re:Don't let them (Score:3, Insightful)

      by TheRealFixer (552803)
      I have to agree. There's very little need for a vendor with a good-working product to have to have that kind of anytime access into your network and servers. And with nebulous new legal requirements like HIPAA (for medical companies) and Sarbanes-Oxley (which the government doesn't even know what it means yet) giving such access to vendors could give you serious trouble in an audit, even with the requisite NDAs and various contracts.

      So, my advice is also to not do it. A vendor needs access to repair/upg
      • Re:Don't let them (Score:2, Informative)

        by RobK (24783)
        I agree with these guys.

        1. Because you're smart enough to ask this question, you've shown that you should be capable of handling the boxes and applications (even if you don't have the time to do it)

        2. The vendor should know their product well enough to give instructions or ask for debug information from you about the configuration/logs and so on.

        3. Do you have the time to fix it when they screw it up?

      • My company makes electronic medical records software. A few months ago it came down the pipe that we were legally required to make a specific modification to the billing codes as of midnight on the last day of the month. The patch only got final approval by our QA department almost immediately before it had to be out in the field.

        Without remote access to our fielded servers, we simply couldn't have pushed it out. We don't have enough people on the deployment team (support isn't trusted with administrative
    • That's all well and good, if management backs you up. On the other hand if management says "We don't care, We need this application, just make it work".

      Management has been known to say things like this even with
      detailed notifacation of exactly what access the vendor is getting
      to the company network.
    • Amen (Score:3, Insightful)

      by jackb_guppy (204733)
      With the internet and remote admin tools coming / have come to full potential, vendors keep coming to ask for this more and more.

      I have been a software vendor for many years. With software that is correctly build and tested, the need have access to a forgien box is about 0.

      That is not say that there were not times I wished I had access. Mainly language translation, English-French with neither tech being able to speak the other and a translator that did not understand techincal terms. Think Baud Rate an
  • My suggestion (Score:5, Insightful)

    by oexeo (816786) on Saturday November 27, 2004 @12:51PM (#10931361)
    Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor.

    What can you suggest?

    Find some better vendors?

    • You know, it's funny... Our customers are notorious for demanding that we can diagnose and fix all problems that occur on the boxes that support our applications. To protect our systems from attacks allowed in by well-meaning but less-than-perfectly-competent customers, we have set up a quarantined network for each customer.

      I'm still looking for better customers :)
  • Two things... (Score:3, Interesting)

    by Sheetrock (152993) on Saturday November 27, 2004 @12:55PM (#10931382) Homepage Journal
    First, as a security-conscious customer you should make your vendor aware of your concerns as well as places where their application violates your security standards. If there are times when their applications require root where it is clearly not necessary, it's a sign that attention may not have been paid to SDP (secure design principles) during the production of the product.

    If a vendor is unsympathetic to your concerns, it's up to you to find an alternative or work around them. As you explain, the second option is not always possible when they require access to a number of services at a fundamental level. The worst cases of this occur when you have one or two vendors to pick from for a given application. My suggestion is then to design the application yourself within your security parameters and functionality requirements -- as many people do not have that capability within their own ASP (otherwise they'd do it already) you might want to use something like Sourceforge and contract a team overseas to do it cheaply, supervising the project from here and optionally open-sourcing it after it's built. Then you've got something designed to your parameters without support or upgrade costs especially if the community digs what you've built.

  • contract (Score:2, Funny)

    by nes11 (767888)
    Can you not just write a contract that holds them completely liable for all damages, losses, downtime, etc caused by that machine? Then give them the option of whether or not they need to be a superuser that badly.
  • Vendors are asshats (Score:4, Interesting)

    by Anonymous Coward on Saturday November 27, 2004 @12:56PM (#10931388)
    Seriously ... . I'm a sysadmin for a medium sized accountancy company, and you wouldn't believe the number of vendors I've had to beat off. I end up talking them down to lower levels of access, all the while listening to the lusers talk down to me as if they knew more about running it on *my* network than the local sysadmin.
    In the long run though, I've figured that these lusers are going to be more trouble than they're worth, and am talking the boss into replacing the NetApps and Junipers (routers) with Gentoo Linux boxen. So in short, don't protect, just reject!
    • Ewww.... (Score:2, Offtopic)

      by caveat (26803)
      ...you wouldn't believe the number of vendors I've had to beat off.

      Well, I hope you at least wash your hands after each one ;)
  • by cybrchld (229583) on Saturday November 27, 2004 @12:56PM (#10931390)
    I have a similar configuration between two in house networks I use the firebox X700 in routing mode it allows me to open only the ports needed to be routed between the networks and also allows me to monitor all traffic to keep an eye on the test network side.
  • VPN (Score:4, Informative)

    by Julian Morrison (5575) on Saturday November 27, 2004 @12:56PM (#10931391)
    Create a dedicated VPN. The misbehaving app server sees only this VPN. Everything it needs to access has a presence thereon, carefully firewalled to allow the relevant ports to open. Everything it doesn't need to access, is not even on the network.
    • The VPN solution is great for the Vendor Company portion of the link, but I would also suggest a "vendor firewall" if you need to extend that service down to machines in the internal network (eg, stores in a grocery chain). The Vendor will *only* see their appserver from the outside, and their appserver will *only* see the vendor's devices internally. Good luck securing the actual end devices; unless you have each device behind a layer 2/3 firewall of some sort. Assume that the vendor can do anything wi
  • service accounts... (Score:2, Informative)

    by Anonymous Coward
    I'm an IT guy who does applications managment for an Air Force network and this is a problem that we have run into as well.

    Exchange Service accounts, NetIQ accounts running as Domain Admins etc.

    This is generally not a good idea, but for your average IT admin superuser domain level rights were the only way to have those programs function properly for the longest time. However, one simply solution that I have just recently tested with our NetIQ application managment Server is to create the local service ac
    • > see which ports have what sort of protocol activity over them. Document the information,

      Why not buy from vendors who list their dependencies properly?
  • VPNS are handy... (Score:4, Informative)

    by eldub1999 (515146) <eldub@pobo[ ]om ['x.c' in gap]> on Saturday November 27, 2004 @01:00PM (#10931419)
    We force the vendors to enter via VPN. we use the VPN gateway to restrict each vendor account's access to only the IP addresses of the systems they need access to. Further, we occasionally use a packet sniffer to watch certain vendors.

    We disable the account by default and require them to contact us and tell us what they are doing (change control) before letting them in.

    Works for us.
  • by Burdell (228580) on Saturday November 27, 2004 @01:01PM (#10931426)
    I work for an ISP, and we do not give anyone outside the system/network administration team root access to anything if at all possible. We have given network vendors (Redback and Ascend/Lucent) remote access to a device a couple of times (we changed the passwords on the particular device before and after), and we have one web application that we had to give the vendor root access for setup (but that was on a dedicated server).

    Our antispam server is a dedicated server, but it still was screwed up once by the vendor. We were having a problem (only half our mail was actually being processed by the filters), and management directed that I go ahead and give the vendor support person access to the server (as the user the software runs under). He logged in to look at it, and within minutes the system was broke and mail was queueing up because the antispam software shut down and wouldn't restart. He had seen something unrelated to our problem that he thought didn't look right and decided to "fix" it for us. As soon as I got him to change it back, I told him to log out and removed his access (and they won't ever get it again).

    After that, the only other time we considered allowing a vendor access was on a problem case that was over a year old that we had to have fixed ASAP. However, in that case, we were NOT going to allow remote access; the vendor was going to have to send somebody to us and we'd sit him down in front of the console (which would be logging) with us looking over his shoulder.

    The only people that know your systems and network are your people (and you'll make enough mistakes). Vendors should not have access to change things at any point; at most they should get a "view" type account (they can look all they want but they can't touch). If something needs changing, they need to tell you and you can make the change (after evaluating it and having a back-out plan). For complex systems, you never end up installing software exactly the way the vendor specified; they are not going to know or understand changes you made for your local configuration (and how such changes affect other services and systems). Even a well-meaning "fix" can cause serious problems, and since it'll be your job to fix them, you better know what was done.
  • (Disclaimer: I was working for Sybase Professional Serives from 95-99)

    You may want to have a look at a Sybase [slashdot.org] product called Replication Server [sybase.com], which permits you to distribute your data in near real time.

    Even though it is not a simple product, setting up a warm standby is fairly straight forward and relatively simple. By setting up appropriate firewall rules you can ensure that the connection is in one direction only. As an added bonus you are better set up in case of a desaster.

    The RDBMS in question

  • by flying_monkies (749570) on Saturday November 27, 2004 @01:27PM (#10931554)
    From the vender side.

    First: It's amazing how many people haven't got a CLUE. "Don't give it to them or get another vender" isn't an option. When you're running a 24x7x365 system and expect sub 1 hour response time from your venders (our customers do) you're going to have to give some.

    All of the boxes we hook up have phone lines into them. If security is an issue on dial-in access, we set it up as follows:

    Modem is enabled dial-out only, no dial in access.
    If the box dials home, we contact the administrator, identify who we are and which box has troubles and have them enable access to the server/hardware and reset the root password temporarily on the box. This occurs only if it's something we haven't already configured to work with sudo.
    We then keep logs of the entire session, then email it to the customer when we're done.
    When we're done, we have them reset the root password and disable dial-in on the modem again.
    If we require access to a network resource from the machine, we're onsite with the customer shoulder surfing.

    We do this at secure military sites, financial institutions, etc. and have yet to have an issue in 20 years.

  • by postbigbang (761081) on Saturday November 27, 2004 @01:31PM (#10931579)
    1) write in to the SLA policy-- a SU agreement with liablity
    2) use (as mentioned) a L7 switch (Telena has one, too)
    3) give temporary access, and make sure you check for root kits everytime with a script
    4) tell your management just how expensive it is to have so many vendor's spoons in the soup and how potentially destabilizing it is to do this
    5) use smart token card access coupled to your own CA; Tie the proximity of the card via RFID to a pacemaker attached directly to the aorta. If they lose it, they die. Simple.
    6) partition roots across servers. Get an SNMP trap when they logon to keep track of them. Set a script against cron to send an additional alarm when they're on for more than a few minutes or upload more than a few megabytes through specific ports (indicating massive changes rather than remote control screen delta)
    7) ask for one of their children for hostage use
  • by Diomidis Spinellis (661697) on Saturday November 27, 2004 @01:41PM (#10931623) Homepage
    [I know this will cost me karma points.]

    The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants. The jail facility is based on the chroot(2) implementation, but prevents well-documented means to escape chroot confinement, offering partitioning of the file system, process, and networking namespaces. The facility removes all super-user privileges that would affect objects not entirely inside the jail.

    For more information read:

    • Poul-Henning Kamp and Robert Watson. Jails: Confining the omnipotent root. In Proceedings, SANE 2000 Conference. NLUUG, 2000.PDF [freebsd.dk], HTML [freebsd.org].
    • Poul-Henning Kamp and Robert Watson. Building Systems to be Shared Securely ACM Queue vol. 2, no. 5 - July/August 2004 HTML [acmqueue.org]
    • The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants.

      This, and other type of chroot environments work great for certain applications, but there is the problem with low-numbered ports in that nobody other than root can create a listener on a port numbered lower than 1024. Many programs that require low level ports (like BIND) don't run as well in a chroot environment because of this. Then the developer has to write what
      • Also I may point out if this vendor uses a Solaris that Solaris zones is supported in Solaris10.

        You can still access ports lower than 1024 without an egg SUID function in a seperate zone running a partitioned instance of Solaris.

        Quite nice.

    • Also, don't forget about these Linux based methods:

      Linux Vserver:
      http://www.linux-vserver.org/ [linux-vserver.org]

      Xen
      http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ [cam.ac.uk]

      User Mode Linux:
      http://usermodelinux.org/ [usermodelinux.org]

      Linux Vserver appears to be almost the same thing as FreeBSD jails. I have not ran either so I cannot speak with authority on them.

      I have ran Xen and UML, however.

      Xen and UML are completely virtualized environments that boot a whole seperate machine inside of the host. UML is good to run on a server th
  • A technique (Score:5, Funny)

    by Gyorg_Lavode (520114) on Saturday November 27, 2004 @01:43PM (#10931645)
    A technuiqe my work employed to get people to stop requesting things is to make some simple form to fill out to get what they want. But then require 2 or 3 signatures. (Their supervisor, their company sponsor or contact and their own.) Then you take 3 or 4 weeks to process any of these forms, (purposefully). And you deny half of them.

    Then you make your policy strictly exclusionary. And when they say "BUT I NEED THIS!", you say, "Ok, fill out a form 23" or whatever the form is. They'll learn quickly that they aren't going to get many of them approved and they'll start putting them in only when they really need them.

    • A technuiqe my work employed to get people to stop requesting things is to make some simple form to fill out to get what they want. But then require 2 or 3 signatures. (Their supervisor, their company sponsor or contact and their own.) Then you take 3 or 4 weeks to process any of these forms, (purposefully). And you deny half of them.


      You've decided to follow the mantra of the BOFH and keep your life simple.

      It's a well-used policy. =)

    • Might as well tattoo "PLEASE OUTSOURCE ME" on your forehead.
  • When the vender begins to sell you their app. Bring up these Issues about not allowing them to have super user access. It is just that simple. When then make them sign a contract that they will not use. And if they begin demmanding then the contract is null and void. Just that simple. Just be smart when buying products. Especially Canned apps.
  • And have them revoke the suppliers rights to sell products for Windows, since they haven't got a clue.

    I am working in a big company, and if the vendor do not spend the little money to fix their broken app, we will pick another. If they haven't got a clue about security on a network, they are are too dumb to deliver software for us.
  • Dude. Run all your apps in a jail (FreeBSD), UML (Linux), virtual machine like VMware (Windows), etc. When they need to be networked, tunnel the connections over SSH. So you'll have several "layers" of network running over the same physical network.

    Say you have an ERP application from one vendor, a CRM application from another, and middleware from another. On each server at your facility, run a virtual machine and run that application in the virtual machine. Connect the virtual machines through SSH tunnels

  • by Anonymous Coward
    ...since VMS has a finer grained privilege system which has no such thing as "superuser" (though heavily priv'd users are possible). In addition there are third party and builtin controls available that allow things like limiting which files/directories/devices even super-priv'd users may get to, may allow certain programs to have privs without the maintenance accounts having them, hiding some storage from programs, etc. etc. Also while a program can be permitted access to kernel space, the privs to do that
  • by TheLibero (750207) on Saturday November 27, 2004 @01:52PM (#10931711)
    Sometimes those a$$holes ask for such things in order to get better view of your infrastructure and update their sales account with information about you. Let me use a non-technical example here. You own a super store that is specialized in Sports clothing. You have a 5-year contract with Nike to supply you with trainers. The contract is gonna expire in 6 months. Also, you have another contract with Puma to supply you with arm bands. Recently, customers have been complaining about the quality of those arm bands, so you contacted Puma (now imagine that in IT world), Puma sales would ask you if they can send a representitive over to check the stock you've got. They do that not only to check the stock, but to check what other products you've been stocking from competitors, so they can update their accounts and have better picture of the areas they have competiton in, and against what companies.

    There is big marketing battle just behind you!

  • by stratjakt (596332) on Saturday November 27, 2004 @02:02PM (#10931764) Journal
    As one of those vendors for governments, I have to constantly deal with some moron admins who refuse to give me any access to the machine, yet CONSTANTLY call for support.

    I don't generally need superuser access on the machine, since generally the only thing that gets screwed up is the data in the RDBMS (you know, user error).

    I had one propellerhead go around in my database deleting tables and columns he felt they didn't need. He told me on the phone "we don't use timestamps here". One of those slam your head on the desk conversations. These are civil servants with lifetime jobs, and maybe they knew all about VAX in 1970, but goddamn if they aren't dense.

    They tend to think that RAID is a magical "never need to backup ever" solution. I just love it when they call me up after their second RAID-5 drive failed, and I ask them when they last did a backup - and they go "uhhhh we don't need to backup we gots RAID".

    Then I explain how RAID has nothing to do with archival or backups, etc, etc.. And I pull out the backup I made last time they had a major upgrade and tell them they have to reenter every parking ticket for the last 8 months, and they threaten and bitch how it's my fault and I tell them I'm not their admin, and if they really want to go to their bosses and fess up how incompetent they are they can go ahead.

    Frankly, I'd love for some more competent clients. Of all of them, I can think of one who has any clue what to do with a computer.

    But then sometimes they call with a problem that requires fixing on the machine. I'm not going to sit on the phone talking them through shit, I'm not going to email them scripts or code, etc. More than once I've had to tell them that if they don't give me access it wont get fixed.

    If it's a problem for you, give them superuser rights when they need it, when they're done doing maintainance, take it away.
  • You don't (Score:3, Informative)

    by bahamat (187909) on Saturday November 27, 2004 @02:04PM (#10931782) Homepage
    You do NOT let outside users have access to production systems as root (or as any other user). When they ask, you tell them to stop smoking crack and make them tell you how to fix it. When they ask again, instead of root, give them the middle finger, and them make them tell you how to fix it.

    You need to learn to do things on your own. Calling up the vendor and handing them root to fix your problems merely makes you vendor dependant, and in my opinion any SysAdmin who does that should be fired. If I were a manager, why would I continue to employ SysAdmins who only call the vendor every time there is a problem? I would be thinking to myself that not only am I paying this bozo a huge sallary, but I'll be paying for the support contract forever as well, and not just for one system, but for however many app servers you've got. Why not just hire someone for minimum wage to dial the vendor when there's a problem than pay someone who is suposedly highly trained but can do nothing for themselves?

    I'm not a manager, but I am the lead SysAdmin for an Internet services company. We have about 40 servers and about half as many networking devices (managed switches, firewalls, load balancers, etc). Whenever we get a new type of device we do end up calling the vendor quite a bit, but we always make sure they teach us the solution and we impliment it. Over six months or so as I learn all the ins and outs and bugs of the device we no longer need to contact the vendor. I have a team of three people, and if one of them gave a vendor the root or enable password on any of our devices I'd have them fired for network security breach, and if they weren't taking proper steps to learn a new device on their own I'd have them fired for incompetence and replaced. There are too many smart people out there to keep employed dumb ones, especially for the price tag SysAdmins deserve.
    • Re:You don't (Score:5, Interesting)

      by MattBurke (58682) on Saturday November 27, 2004 @03:03PM (#10932176)
      You obviously can't have ever dealt with anything of importance then...

      If you have something go down on you which is costing several people's salaries per hour in lost revenue, you don't sit around for days trying to sus it out - you get the vendor to fix it. That's why you pay them for their support. You do pay for vendor support don't you?

      They know their products a hell of a lot better than you ever will, and you'll likely have a contract saying it will be fixed in a stated timescale or they'll be paying you compensation. Often vendors will courier out replacement hardware instantly before the engineers have even sat down to log into the device, and those engineers will be of the highest quality if you've flagged a critical call. If you've triggered a bug in a piece of vendor kit, how long will it be before you've diagnosed it to be a bug? Then how long will it be before you get a fix? I know of at least one large company which has compiled and installed a new software image on a customer device within hours of a critical fault call being raised. Would they do that if you weren't paying for support? I think not.

      For maintainence however, you learn how to do it yourself.
  • by scorp1us (235526) on Saturday November 27, 2004 @02:11PM (#10931829) Journal
    Let me qualify explain that statement. I support a small web-based application. It'll win on Unix or Windows (under Apache & PHP) I don't care about that stuff. But what does piss me off is when we have to port the app to every DB MS under the sun. Our support costs go through the roof just because the customer wants it their app DB.

    Well, try explaining to IT (not a smart bunch, after all they dropped out of CS (or never even tried)) that their DB du jour is not up to the task of our RDBMS. True 90% SQL compliance is a good start, but there it is impossible to create a full-featured app and still be completely agnostic.

    Examples: MySQL is the worst. The last insert ID is a horrible concept that is not portible. Try finding it in Informix, it is impossible to find, but exists. Thent here is the MySQL timestamp 'feature' - the first timestanp field (and only the first) when UPDATEd is updated to NOW() regardless of whether or not you include it in the updated feilds.

    The only reason why they want the DB changed is because they have lazy IT departments that want to do nothing. Their IT staff does not understand the complexities of SQL DBs to them it is just an DB, and "thay're all compabible through SQL" yeah, right. "What about ODBC?" ODBC is fine for basic record operations, but the moment you try to do anyhing advanced, you're out of luck.

    I used MySQL as an example, but I've worked with several other Databases, and there is no Holy Grail of DB abstraction. There's much more to databases than inserting updating, deleting and selecting rows, depsite what IT thinks.

    • I realize that my tiraid may not have been completely explained. We have no problem installing our required DBMS on whatever server or OS (customer's or provided by us), but so many times they will not install our DBMS on their production servers, so we end up with our own app server.

      This is not my fault, this is theirs. They want to take no risks with disrupting other things, so the price of that is another server. They bring it on themselves.

      My angst is directoed towards the ones who will not install ou
    • I was going to recommend ODBC or ADO's and port it to win32...gasp.

      I would just explain the costs involved and offer to port it to their database but charge an extravagant fee for the development. After that then say use database A since this is what my program is designed to use and it will be hell of alot cheaper. But if you want to spend more money that is up to you.

    • it is impossible to create a full-featured app and still be completely agnostic

      Not to mention that if you want the application to perform well you must use vendor specific extensions.

  • Depending on the nature of the application, you can give the vendor a UML [sourceforge.net] machine and its root access. This lets them do what they want/need to, without compromising your environment.
  • by Eneff (96967) on Saturday November 27, 2004 @02:19PM (#10931879)
    1. Smaller niche vendors usually don't have the hardware or manpower to simulate your system. You went with them because they were half the price of dealing with IBM, remember? :)

    2. Often, those applications that require Superuser for the installation are third party applications. Install it yourself. You may have more trouble on the outset, but you'll know what's going on with your machine.

    3. Often, vendors have their programmers as installers. The bad news is that you see the problems you do with the installations. The good news is that they'll know exactly why they need root - and they'll tell you what they need done. This might need to be a tag team installation, of course.

    4. Remember, you can always invite them up there if you want to pay them for their time. Remote access is requested because it's cheaper. Alternatively, put in the contract that you want installation instructions. It will take more time, but you're always welcome to pay more.

    Many of these vendor problems are reduced to cost-cutting measures. If you want to pay more, then vendors will be willing to oblige.
  • For somebody who is simply selling you software to require that level of access to a server - ANY server - is illogical and a probable security hole. Might I suggest using the wheel?
  • I'm admin for a optical shop and when they asked for root, thats just what I told him.. I also said "That IF I EVER allow you root, it will be through remote secure desktop and I WILL be watching your every move! AND, if you do get stupid with that mouse I will pull the plug and call your boss, is that clear??"

    They quickly agreed, since I got layered security in place (router, software, NAT, etc..), they'll have the devil's own time just making the attempt since it'll take me an half hour just to configur
  • Back when I started, working on mainframes, the standard, everywhere, was that you had a development environment, and when you were done making your changes, you passed it over to the admins, who moved it into a test environment that replicated the production environment (often with less data, but otherwise identical), ran regression tests, and *then* put it into production.

    Oh, but that's mainframes, where they run whole companies, not workstations/servers that, uh, oops... Well, maybe you could lower your
  • by Bishop (4500) on Saturday November 27, 2004 @02:47PM (#10932053)
    Your vendors can already run arbitrary code on your systems. You must trust them.

    The standard practice of firewalls to restrict access, and lower priviledge accounts are all good (and important). But the proper protection should have been negotiated into the service contract in terms of access the vendor requires and liability. Durring that negotiation process the technical authority should have considered the security concerns (and added costs). If the technical authority was inept the best you can do is minimize the risks now, and use this example to raise security concerns for the next contracts.

  • If its a unix apps there probably is a Solaris port. If its a Linux app you can run it under Solaris10 x86 in a seperate zone.

    Similiar to FreeBSD Jails you can totally isolate the process but I think Solaris Zones partition a whole instance of the OS which gives it more functionality than BSD Jails. For example you can access ports lower than 1024 as root but in a seperate Solaris partition.

    I am a little ignorant about this since I have just been reading about it. Anyone here know more info?
  • An easy solution? (Score:3, Interesting)

    by bigberk (547360) <bigberk@users.pc9.org> on Saturday November 27, 2004 @02:56PM (#10932121)
    What if you set up your Linux system with User Mode Linux [sourceforge.net], or your FreeBSD system with FreeBSD jails [freebsd.org], like modern hosting companies provide. This will provide your external customer/vendor with root access, but within a locked in virtual server. If your external vendor wants to maintain their database installation, fine: they have root on their own "machine", on their own IP, and there is very little risk to the larger system with real root.
  • by JoeShmoe (90109) <askjoeshmoe@hotmail.com> on Saturday November 27, 2004 @03:39PM (#10932385)
    May not apply to *nix but for Windows servers, I answer requests like this with Remote Control on a Terminal session. If a vendor needs to get "Administator" level access for some reason, I push them through Terminal Services then remote control their session so I or someone else can watch what they do. If the vendor is smart (haven't found one yet) they are aware that someone can watch what they are doing, but they aren't so I usually get to watch them do all kinds of boneheaded things while they take a guess and check approach to fixing a problem. The downside for me is I have to take notes so when the vendor finally disconnects I can undo anythign they did that was outside the scope of their repair.

    Which makes me wonder why I can't record an RDP session somehow. This would serve two purposes: a) it would let me replay what a vendor had done later when it is more convenient and b) it would give me some proof if I later have to go blast a vendor for doing something absurdly stupid.

    The same question for VNC or really any kind of remote management tool that is likely to be used on the Windows platform. Can any of them be logged and/or replayed?

    - JoeShmoe
    .
    • Which makes me wonder why I can't record an RDP session somehow.
      It may not be the solution you're looking for, but there's several utilities out there that can record the activity of a window (i.e. the RD window) and make a movie out of it.
      These serve mainly to create demos and stuff like that.

      Hope it helps.
  • I develop and code a web-based database application, that is installed on a dedicated server at our client's sites. In most cases, we also install some kind of remote access software for maintainance. We access this software via Modem or ISDN with or without callback, VPN or special firewall configurations via internet. Some clients even expose the server to the internet.

    • Some clients (usually smaller companies and non-profit organisations) have very low security standards and trust us to be "the good ones
  • 1) First off, it amazes me how many customers there are who don't have security. For example, If I just have a user account on your system on your network - I still have access to your whole network unless you specifically firewall me off.

    2) I prefer be firewalled off, so I know their stuff won't mess with ours and our stuf won't mess with theirs - but the truth is most customers don't want to go thru that effort or cost. Also, unfortunately, most have no clue what their doing other than following a check
  • of all the answers, here, i'm surprised that none of them mention these two projects:

    http://sf.net/projects/selinux

    http://sf.net/projects/xen

    the first project allows you to grant root access to lusers, thereby convincing the program that it's got root access, but the SELinux security kicks in as well, which is far more flexible than the 20+-year-old unix security model, and most importantly SELinux doesn't give a rats arse about what a superuser is.

    the second project, xen, is like vmware only faster and
  • Solaris 10 has a very neat feature that lets you configure zones on a system. Each zone can run a specific service or applications and communicate via the network using its own designated IP address. In side the zone there is no way to break out into the rest of the system. Even with root access in that zone.

    You would still want to take additional precautions if you let outsiders on your systems even in a zone. Monitor what is done and record all changes.
  • We have a DMZ between two Raptor firewalls. Applications that need access will either get a hole punched through the firewall for the particular port (IP limited) or if that's not sufficient / too complicated, we setup a VPN for the application. FWIW, we use Nortel Conntivity to do it but honestly I don't think there's much advantage to using a vendor vs an open source solution...
  • "We've painted ourselves into a corner and, since we can't think of a way out, we're asking Slashdot for free technical advice to further our commercial goals without actually paying any of you."

    "Hey do you think they bought it?"
  • In my company, except for the possibility of a rare manager being that innept, we wouldn't tolerate it. Computer security isn't always a show stopper, but allowing external access to machines with company data almost always is. If nothing else, legal would come down hard and say no.

    So my advice would be to go explain it to the lawyers. If it scares them (meaning if you communicate effectively why is scares you) they might have enough clout to kill the deal, causing others to try to convince the vendor

"Never give in. Never give in. Never. Never. Never." -- Winston Churchill

Working...