Protecting Your Enterprise Network from Vendor App Servers? 258
anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?"
"Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.
Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'
What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"
Layer 7 Firewall (Score:5, Informative)
L7 Firewalls usually get a bad rap because they tend to be pretty fussy in setup, something you can't really avoind with this kind of stuff. Also, if I was in your shoes, I would learn to stop worrying and start loving tight-ass SLA's...
Re:Layer 7 Firewall (Score:2, Interesting)
It's a pitty they don't teach this stuff in CS:
http://www.dyadsecurity.com/papers/rbac.html [dyadsecurity.com]
http://www.nsa.gov/selinux/papers/inevit-abs.cfm http://www.acm.org/classics/sep95/ rtsp://media-1.datamerica.com/blackhat/bh-usa-00/
Re:Layer 7 Firewall (Score:4, Informative)
As far as Mandatory Access Control (MAC) goes, it is even more difficult to manage than an AP firewall, and a terrible pain to implement - ever tried to make a MAC model work in an Active Directory environment? Not easy...
And even if you just choose to implement the RBAC part of a MAC model, how do you define roles? Unless you have a very stratified and well-defined role structure for the people who work in the enterprise, it is a daunting task to set up roles. In one place I worked, there were ~10,000 employees - and 4800+ job titles. Not exactly conducive to role assignment, to be sure.
Re:CISSP Award for Excellence (Score:2)
Consultancy (Score:5, Funny)
Re:Consultancy (Score:2)
AAARGH (Score:2)
Tell them to screw off.. (Score:5, Informative)
I built a network for vendor lab equipment that was it's own netowrk, connected to the main network via a file server with 2 nics and NO routing. Therefore, file could be transfered from the lab network to the file server, and then from there to any of the main networks, of course after they had been scanned for viruses and such, since anti virus software usually would not run on their equipment. My rules were firm and backed by higher-ups - if you couldnt get your equipment to work in the enivroment given, or with only minimal flexibility from me (for example i'd write scripts to move their files automatically to the main, backed up network or something simple like that) - then we will find another vendor to provide us a solution that fits. Period. I never had any problem with a vendor once they heard the terms. They won't make money off us if they can't make the system work in our environment.
Hell, yes. (Score:2)
Our integration components are likewise designed to be flexible and nonintrusive -- as much code as possible on our server, as little as possible on the system we're integrating with.
I'd like to think that when we start deploying to larger environments, these efforts will get noticed.
Re:Hell, yes. (Score:2)
Re:Hell, yes. (Score:2)
How about an asinine suggestion?
The icon on the desktop is a link to a
The html file tries via JS to load two (hidden) images, one from each address.
Whichever image's onload event triggers first causes a document.replace() with the full site URL corresponding to that image.
Another option, if you're already tied to IE, would be to have an HTA/ActiveX control check the machine's IP number and make a decision from there.
Or, if they're already going through a proxy, use proxy autoconfiguration/name resolution to sort the mess out.
Lots of solutions out there.
Re:Tell them to screw off.. (Score:2)
Yea, been there.
Re:Tell them to screw off.. (Score:5, Insightful)
-- Lee K. Gleason, VMS sysadmin
Re:Tell them to screw off.. (Score:3, Interesting)
The manager needs to have the password in a safe someplace so that if you all quit at once he can get someone else to keep the systems going.
Otherwise, whats the problem with giving him root? Just because says tradition is that account 0 one unix systems is called root doesn't mean yours has to be called root. Rename the root account to something else, and create a no privliges account called root that you can give anyone the password to.
If nothing else that is a good way to weed out those who need higher access. The people who don't figure it out right away are either too incompitent to have account, or can be trusted to never even look at root until it is really needed. (which is left as an exercise to the user) Those who notice right away have enough of a clue that they might not break something.
Note that on a properly setup system you do not need a root password at all. sudo can run all the programs you need. At least that is what I'm told, I've never got my systems that far.
Re:Tell them to screw off.. (Score:4, Insightful)
Switch vendors (Score:5, Informative)
There are a few things you can do in this case:
1. Get a different vendor (if the application truly does require su for vendor support).
2. Reqest different support staff from the vendor (do this if the app doesn't require su but the support staff is too lazy).
3. Learn how to support the app in-house
I am very serious on these points. You do not want to give root access to people that should not need it. If they say they need it, they have an awful application.
The place I work has a vendor area fenced off from the rest of the server room, and the vendors only have access to what they need. If they need more privileges, the IT guys watch them like a hawk until they're done.
Re:Switch vendors (Score:3, Informative)
Re:Switch vendors (Score:3, Informative)
At least, it used to... Luckily I haven't been the poor sod to do installs lately.
(I do remember the first time I had to install ArcIMS... what a piece of hell.)
Re:Switch vendors (Score:2)
Sure, lock them out. You have a problem with their software, call their tech support, and when they ask for access, tell them to go fish... but then don't bitch about the callout charge to get an engineer out to your site.
Your call.
Re:Switch vendors (Score:5, Interesting)
I've worked escalation support for a large storage vendor for a number of years, and while I completely understand (and agree with!) limiting vendor access, I find that IT departments tend to be *very* unresponsive when the vendor support folks need their help.
It's not exactly efficient, for example, when I have to wait 7 hours for logs to be uploaded, 5 hours for the "network guy" to respond to a page and fix the duplex mismatch he created that's causing 50% of packets from our systems to clients to be dropped, weeks for the system admin to stop piddling around and get Sun on the phone when we have valid interoperability issues (Sun won't talk to most other vendors except through the common customer), etc.
In the meantime, in my experience, the CIO tends to lambast the vendors for "poor responsiveness" and "terrible support" - though the smart ones eventually realize that the IT staff is shooting themselves in the foot, and do something about it.
IT departments that are responsive to vendors, on the other hand, are a breeze to work with, and issues are typically resolved many times faster.
Re:Switch vendors (Score:5, Insightful)
When the only alternative to required software is working by hand (or a major reverse-engineering project), you just gotta suck it up and figure out how to protect the rest of your systems from their arrogance.
Generally speaking, it's a clear sign of laziness or incompetence on the part of a half-assed programmer to think he needs root for everything. Hell, Oracle doesn't need root to run, and it's a mighty damned complicated RDBMS suite. If you're stuck with one of these vendors, I urge you to make it clear to them that you are using their software because you are stuck with it, that given the chance you will jump ship in a heartbeat, and that the reason you'll never buy any of their other products is that their claim to "need" root is a sign of either ineptitude or a cavalier attitude towards their customers.
Re:Switch vendors (Score:2)
Generally speaking, it's a clear sign of laziness or incompetence on the part of a half-assed programmer to think he needs root for everything.
Amen. The OP said that his company had a large IT infrastructure which means they are worth the vendors business. I urge him to apply whatever pressure he can on the vendors to change their ways.
That way, little places like where I currently work might get the same benefit!
Re:Switch vendors (Score:4, Interesting)
That where the vendor's software is on a "black-box" machine that only they have administrative access to, and which runs no other software.
I'm (senior deployment engineer at) one of those vendors. Not only is our app fairly complex to administer, but every server that goes out contains a copy of the "secret sauce" -- not our application itself (which our bigger competitors could probably recreate in 9 months or less), but the data behind it (much more expensive and difficult to recreate). Consequently, our management is paranoid -- the servers are rigged to self-destruct (wipe the keys that allow them to decrypt the partition where all data is stored) in the event that any attempted tampering is detected, and can only be reenabled by the very small subset of our staff w/ access to the private keys.
Right now our app has components that run as 4 different users (to isolate breakins to the component where they occur), and includes a firewall and a VPN solution (both of which need root for obvious reasons).
Since it's our box that's running the app, and that same system does nothing else -- why restrict our access further? Absolutely, firewall us off from anything we don't need -- but restricting access to our box seems silly (and is something we'd consider only for a very large customer -- which is just as well, since the small folks we're selling to right now haven't had a problem).
In any event, I'm curious to hear how you'd respond.
Re:Switch vendors (Score:2)
I just can't see where a black-box app vendor would ever need continuing access to root on customer boxes. As many others have pointed out, if the vendor needs root access for a specific task, then it's fairly easy to set up sudo for their needs; however, giving someone the keys to the kingdom just because they might need them at some point in the future is foolish at best.
Re:Switch vendors (Score:2)
No (Score:2)
You're thinking of HIPAA.
Sox stands for Sarbanes-Oxley Act [google.com]. It applies to every publicly-traded US company.
Re:Switch vendors (Score:2)
make sure everyone understands what the problem is (Score:3, Informative)
On demand and via a process (Score:2, Insightful)
The disadvantage is that there is an operational overhead -- but then there should be because if its a pain to change things then less gets changed.
Re:On demand and via a process (Score:3, Insightful)
What if the problem is caused by bogosity in a config file that only root can read?
What if the logs produced by the application are only readable by root (or by adm)?
What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?
Re:On demand and via a process (Score:3, Insightful)
Then it's written wrong. I want the config files to have permissions for a user named $vendor.
What if the logs produced by the application are only readable by root (or by adm)?
Then it's written wrong. I want the log files written to a location for a user named $vendor.
What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?
Then it's written wrong. I want the application to run in a chroot or jailed environment limited $vendor and $vendorapp.
There is *no* good reason for a vendor app to run as root/superuser/Administrator. It's poor coding, and poor design. If that's the best your 'cheaper' vendor can do, you'll end up paying more in the long run for support, security problems, and related issues. It's too easy "since you're root" to just change file permissions to make your app 'just work', instead of correcting the application to work within the system constraints.
Re:On demand and via a process (Score:2)
What if the problem is caused by bogosity in a config file that only root can read?
What if the logs produced by the application are only readable by root (or by adm)?
What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?
All of these can be handled with a proper sudo configuration. sudo rights to cat the log files and config files, you vendor does know which files he needs to read right? I supervise a team that supports over 1000 servers mostly running AIX, Red Hat and Solaris and some of the sudo config files are close to 1000 lines. Why because no one but the SA's have sudo su -, everyone else must supply the commands they will need to run as root ahead of time. Granted sudo isn't the full answer but it is part of it after you have a secure but limited method for vendors to connect. Now giving write access is a bit harder because you don't want to allow things like vi since you can escape out to a root shell. On one of my accounts rvi is deployed which doesn't allow shell commands to be run from vi, reducing the risk.
Don't let them (Score:5, Informative)
Simple answer: Don't let them. This is standard operating procedure in the financial services industry. Do you think that a bank would EVER let a non-employee or non-contractor access any bank system whatsoever? Especially remotely? If these companies want to do business with you, they WILL play by your rules or you'll pick one of their competitors products. In my experience, the only companies that required remote access to their systems were ones that A. Didn't have a fully working product, and B. Had to have the developer log in constantly to patch binaries with the latest bug fixes just to get a semi-working product. These are not the type of companies you want to do business with in the first place.
Speaking as a sysadmin for a smallish financial services company that processes around 1.2 million transactions a month, I would NEVER allow any vendor remote access to our network. It just wouldn't happen. Even if I did want to give them access, there are rules and regulations that strictly forbid my giving them access. If they give you a hard time, make something up about a security audit or something.
Seriously, you are asking for trouble if you let them have access. Who's going to take the blame when one of their developers logs in and wipes out all of your company's data? Chances are they'll blame it on you and you'll be in trouble for their mistake.
Re:Don't let them (Score:3, Insightful)
So, my advice is also to not do it. A vendor needs access to repair/upgrade a system? Arrange for a once-off remote access solution (like WebEx or something for Windows boxes) in a situation where you have control of the access and can monitor them. We had a vendor one time get real whiney about not having on-demand remote access into our system, but we put out foot down and they dealt with it. The reality is that you're the customer, and it's their job to adapt to your situation.
Re:Don't let them (Score:2, Informative)
1. Because you're smart enough to ask this question, you've shown that you should be capable of handling the boxes and applications (even if you don't have the time to do it)
2. The vendor should know their product well enough to give instructions or ask for debug information from you about the configuration/logs and so on.
3. Do you have the time to fix it when they screw it up?
Not always good enough. (Score:2)
Without remote access to our fielded servers, we simply couldn't have pushed it out. We don't have enough people on the deployment team (support isn't trusted with administrative access on the fielded systems, though we give them access to monitoring tools) to log into all our fielded systems and run the update manually, nor would it be as consistant as remotely applying an automated patch that's been approved by our QA department.
Mind you, when I say "our fielded servers", I mean it. We own those systems (though our customers paid for them), we have sole administrative access to them, and they run no other software. Perhaps your situation's different.
Re:Don't let them (Score:2)
Management has been known to say things like this even with
detailed notifacation of exactly what access the vendor is getting
to the company network.
Amen (Score:3, Insightful)
I have been a software vendor for many years. With software that is correctly build and tested, the need have access to a forgien box is about 0.
That is not say that there were not times I wished I had access. Mainly language translation, English-French with neither tech being able to speak the other and a translator that did not understand techincal terms. Think Baud Rate and Partity Bit
Even when I went on site, I generally never touched a machine. I always had the local operator or manager do the actual typing. This way, they learned the what and how along with me, and how to fix it.
But back to today's world, IF and I MEAN IF the higher levels of company force/black mail you into a corner , setup a connection PC. Vendor connects to there, and that is terminal of them in your system. Log every thing. Turn off the PC when you can not be watching what happening.
My suggestion (Score:5, Insightful)
What can you suggest?
Find some better vendors?
Re:My suggestion (Score:2)
I'm still looking for better customers
Two things... (Score:3, Interesting)
If a vendor is unsympathetic to your concerns, it's up to you to find an alternative or work around them. As you explain, the second option is not always possible when they require access to a number of services at a fundamental level. The worst cases of this occur when you have one or two vendors to pick from for a given application. My suggestion is then to design the application yourself within your security parameters and functionality requirements -- as many people do not have that capability within their own ASP (otherwise they'd do it already) you might want to use something like Sourceforge and contract a team overseas to do it cheaply, supervising the project from here and optionally open-sourcing it after it's built. Then you've got something designed to your parameters without support or upgrade costs especially if the community digs what you've built.
contract (Score:2, Funny)
Vendors are asshats (Score:4, Interesting)
In the long run though, I've figured that these lusers are going to be more trouble than they're worth, and am talking the boss into replacing the NetApps and Junipers (routers) with Gentoo Linux boxen. So in short, don't protect, just reject!
Ewww.... (Score:2, Offtopic)
Well, I hope you at least wash your hands after each one
Simple use a Firewall (Score:3, Insightful)
VPN (Score:4, Informative)
Additionally (Score:2)
service accounts... (Score:2, Informative)
Exchange Service accounts, NetIQ accounts running as Domain Admins etc.
This is generally not a good idea, but for your average IT admin superuser domain level rights were the only way to have those programs function properly for the longest time. However, one simply solution that I have just recently tested with our NetIQ application managment Server is to create the local service account on say...our SMS server not as a domain admin, but just as a local admin. If I do this I can avoid giving blanket access to the entire domain to that program, and I can control which servers have that particular service account on them.
Obviously its not a total fix, although luckily for us, server 2003 has eliminated a lot of the most common "null session" issues that were such a pain to hunt down and change manually.
The only other thing I could suggest is creating a test-bed network and installing one of the suspect application servers at a time and using a sniffer capture program to see which ports have what sort of protocol activity over them. Document the information, test several application servers at a time, shut down uneeded ports and then implement it on your network. *with approval of course
I do hope that I haven't completely misunderstood your problem.
Re:service accounts... (Score:2)
Why not buy from vendors who list their dependencies properly?
VPNS are handy... (Score:4, Informative)
We disable the account by default and require them to contact us and tell us what they are doing (change control) before letting them in.
Works for us.
Re:Doesn't solve the problem (Score:2)
They can screw up without root (Score:5, Interesting)
Our antispam server is a dedicated server, but it still was screwed up once by the vendor. We were having a problem (only half our mail was actually being processed by the filters), and management directed that I go ahead and give the vendor support person access to the server (as the user the software runs under). He logged in to look at it, and within minutes the system was broke and mail was queueing up because the antispam software shut down and wouldn't restart. He had seen something unrelated to our problem that he thought didn't look right and decided to "fix" it for us. As soon as I got him to change it back, I told him to log out and removed his access (and they won't ever get it again).
After that, the only other time we considered allowing a vendor access was on a problem case that was over a year old that we had to have fixed ASAP. However, in that case, we were NOT going to allow remote access; the vendor was going to have to send somebody to us and we'd sit him down in front of the console (which would be logging) with us looking over his shoulder.
The only people that know your systems and network are your people (and you'll make enough mistakes). Vendors should not have access to change things at any point; at most they should get a "view" type account (they can look all they want but they can't touch). If something needs changing, they need to tell you and you can make the change (after evaluating it and having a back-out plan). For complex systems, you never end up installing software exactly the way the vendor specified; they are not going to know or understand changes you made for your local configuration (and how such changes affect other services and systems). Even a well-meaning "fix" can cause serious problems, and since it'll be your job to fix them, you better know what was done.
RDBMS (Score:2)
You may want to have a look at a Sybase [slashdot.org] product called Replication Server [sybase.com], which permits you to distribute your data in near real time.
Even though it is not a simple product, setting up a warm standby is fairly straight forward and relatively simple. By setting up appropriate firewall rules you can ensure that the connection is in one direction only. As an added bonus you are better set up in case of a desaster.
The RDBMS in question need not to be Sybase ASE. It works fine with just about any major RDBMS. In fact: There are Sybase customers that use Rep Server in order to replicate from Oracle to Oracle, since Oracle "Replication" just plain sucks!
I go through this all of the time (Score:5, Informative)
First: It's amazing how many people haven't got a CLUE. "Don't give it to them or get another vender" isn't an option. When you're running a 24x7x365 system and expect sub 1 hour response time from your venders (our customers do) you're going to have to give some.
All of the boxes we hook up have phone lines into them. If security is an issue on dial-in access, we set it up as follows:
Modem is enabled dial-out only, no dial in access.
If the box dials home, we contact the administrator, identify who we are and which box has troubles and have them enable access to the server/hardware and reset the root password temporarily on the box. This occurs only if it's something we haven't already configured to work with sudo.
We then keep logs of the entire session, then email it to the customer when we're done.
When we're done, we have them reset the root password and disable dial-in on the modem again.
If we require access to a network resource from the machine, we're onsite with the customer shoulder surfing.
We do this at secure military sites, financial institutions, etc. and have yet to have an issue in 20 years.
Several different possible solutions (Score:3, Funny)
2) use (as mentioned) a L7 switch (Telena has one, too)
3) give temporary access, and make sure you check for root kits everytime with a script
4) tell your management just how expensive it is to have so many vendor's spoons in the soup and how potentially destabilizing it is to do this
5) use smart token card access coupled to your own CA; Tie the proximity of the card via RFID to a pacemaker attached directly to the aorta. If they lose it, they die. Simple.
6) partition roots across servers. Get an SNMP trap when they logon to keep track of them. Set a script against cron to send an additional alarm when they're on for more than a few minutes or upload more than a few megabytes through specific ports (indicating massive changes rather than remote control screen delta)
7) ask for one of their children for hostage use
Check-out the FreeBSD jail facility (Score:5, Informative)
The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants. The jail facility is based on the chroot(2) implementation, but prevents well-documented means to escape chroot confinement, offering partitioning of the file system, process, and networking namespaces. The facility removes all super-user privileges that would affect objects not entirely inside the jail.
For more information read:
Re:Check-out the FreeBSD jail facility (Score:2)
This, and other type of chroot environments work great for certain applications, but there is the problem with low-numbered ports in that nobody other than root can create a listener on a port numbered lower than 1024. Many programs that require low level ports (like BIND) don't run as well in a chroot environment because of this. Then the developer has to write what is known as an egg or a function that executes with SUID privileges and "hatches" in order to allow access to the lower numbered port. This opens up another security hole because now the SUID function or egg can be exploited to allow a remote attacker to gain root or elevate privileges.
So I will agree with you that good system administration practice requires that you run a lot of processes in a chroot jail, however there are a lot of cases where this isn't possible, especially with closed-source commercial applications.
Re:Check-out the FreeBSD jail facility (Score:2)
You can still access ports lower than 1024 without an egg SUID function in a seperate zone running a partitioned instance of Solaris.
Quite nice.
Re:Check-out the FreeBSD jail facility (Score:2)
Linux Vserver:
http://www.linux-vserver.org/ [linux-vserver.org]
Xen
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ [cam.ac.uk]
User Mode Linux:
http://usermodelinux.org/ [usermodelinux.org]
Linux Vserver appears to be almost the same thing as FreeBSD jails. I have not ran either so I cannot speak with authority on them.
I have ran Xen and UML, however.
Xen and UML are completely virtualized environments that boot a whole seperate machine inside of the host. UML is good to run on a server that has some 'extra' resources to dedicate to a VM as you can spec total ram on the command line and there are no modifications to the host machine kernel. Xen's hypervisor takes over the machine by only occupying the first 64-128M of ram, with the rest of ram dedicated to VMs, which makes it hard, but not impossible, to slap Xen onto an existing machine. Xen also requires that a patched kernel is installed on the host.
-ft
A technique (Score:5, Funny)
Then you make your policy strictly exclusionary. And when they say "BUT I NEED THIS!", you say, "Ok, fill out a form 23" or whatever the form is. They'll learn quickly that they aren't going to get many of them approved and they'll start putting them in only when they really need them.
So .... (Score:2)
You've decided to follow the mantra of the BOFH and keep your life simple.
It's a well-used policy. =)
Re:A technique (Score:2)
Get them before you buy the application. (Score:2)
Call Microsoft (Score:2)
I am working in a big company, and if the vendor do not spend the little money to fix their broken app, we will pick another. If they haven't got a clue about security on a network, they are are too dumb to deliver software for us.
How to do it. (Score:2)
Say you have an ERP application from one vendor, a CRM application from another, and middleware from another. On each server at your facility, run a virtual machine and run that application in the virtual machine. Connect the virtual machines through SSH tunnels that run from virtual machine on one physical machine to virtual machine on another physical machine.
This will increase processing time, necessary hardware, administration, etc., in other words, COST, but this can be considered a necessary security cost.
In the meantime, I would get a software development department, organize a SourceForge project for each application you guys need, or join an existing such project, and implement in-house free software versions of the same things. You can rest assured that other businesses have similar needs, and these free software projects will eventually replace the expensive vendor controlled versions that you run now. This will decrease costs in the long run.
Let them run under VMS (Score:2, Interesting)
If you need to run more safely in x86, I would suggest having a look at SELinux (from NSA) which gives some better controls than usual. Pay special attention to protections of devices, especially some of the more dangerous ones like
Non-technical reasons! (Score:3, Interesting)
There is big marketing battle just behind you!
heh, my experience is the opposite (Score:5, Insightful)
I don't generally need superuser access on the machine, since generally the only thing that gets screwed up is the data in the RDBMS (you know, user error).
I had one propellerhead go around in my database deleting tables and columns he felt they didn't need. He told me on the phone "we don't use timestamps here". One of those slam your head on the desk conversations. These are civil servants with lifetime jobs, and maybe they knew all about VAX in 1970, but goddamn if they aren't dense.
They tend to think that RAID is a magical "never need to backup ever" solution. I just love it when they call me up after their second RAID-5 drive failed, and I ask them when they last did a backup - and they go "uhhhh we don't need to backup we gots RAID".
Then I explain how RAID has nothing to do with archival or backups, etc, etc.. And I pull out the backup I made last time they had a major upgrade and tell them they have to reenter every parking ticket for the last 8 months, and they threaten and bitch how it's my fault and I tell them I'm not their admin, and if they really want to go to their bosses and fess up how incompetent they are they can go ahead.
Frankly, I'd love for some more competent clients. Of all of them, I can think of one who has any clue what to do with a computer.
But then sometimes they call with a problem that requires fixing on the machine. I'm not going to sit on the phone talking them through shit, I'm not going to email them scripts or code, etc. More than once I've had to tell them that if they don't give me access it wont get fixed.
If it's a problem for you, give them superuser rights when they need it, when they're done doing maintainance, take it away.
You don't (Score:3, Informative)
You need to learn to do things on your own. Calling up the vendor and handing them root to fix your problems merely makes you vendor dependant, and in my opinion any SysAdmin who does that should be fired. If I were a manager, why would I continue to employ SysAdmins who only call the vendor every time there is a problem? I would be thinking to myself that not only am I paying this bozo a huge sallary, but I'll be paying for the support contract forever as well, and not just for one system, but for however many app servers you've got. Why not just hire someone for minimum wage to dial the vendor when there's a problem than pay someone who is suposedly highly trained but can do nothing for themselves?
I'm not a manager, but I am the lead SysAdmin for an Internet services company. We have about 40 servers and about half as many networking devices (managed switches, firewalls, load balancers, etc). Whenever we get a new type of device we do end up calling the vendor quite a bit, but we always make sure they teach us the solution and we impliment it. Over six months or so as I learn all the ins and outs and bugs of the device we no longer need to contact the vendor. I have a team of three people, and if one of them gave a vendor the root or enable password on any of our devices I'd have them fired for network security breach, and if they weren't taking proper steps to learn a new device on their own I'd have them fired for incompetence and replaced. There are too many smart people out there to keep employed dumb ones, especially for the price tag SysAdmins deserve.
Re:You don't (Score:5, Interesting)
If you have something go down on you which is costing several people's salaries per hour in lost revenue, you don't sit around for days trying to sus it out - you get the vendor to fix it. That's why you pay them for their support. You do pay for vendor support don't you?
They know their products a hell of a lot better than you ever will, and you'll likely have a contract saying it will be fixed in a stated timescale or they'll be paying you compensation. Often vendors will courier out replacement hardware instantly before the engineers have even sat down to log into the device, and those engineers will be of the highest quality if you've flagged a critical call. If you've triggered a bug in a piece of vendor kit, how long will it be before you've diagnosed it to be a bug? Then how long will it be before you get a fix? I know of at least one large company which has compiled and installed a new software image on a customer device within hours of a critical fault call being raised. Would they do that if you weren't paying for support? I think not.
For maintainence however, you learn how to do it yourself.
Re:You don't (Score:2)
I'll bet they've got one motherf***er contract with their vendors too. If the shit hits the fan, I'll bet their own admins would probably get the sack instantly for even logging into one of their systems unless under the direction of the vendor!
Put short, it's a situation close to outsourcing, except maintainence is done internally.
As a vendor, customers are idiots (Score:3, Interesting)
Well, try explaining to IT (not a smart bunch, after all they dropped out of CS (or never even tried)) that their DB du jour is not up to the task of our RDBMS. True 90% SQL compliance is a good start, but there it is impossible to create a full-featured app and still be completely agnostic.
Examples: MySQL is the worst. The last insert ID is a horrible concept that is not portible. Try finding it in Informix, it is impossible to find, but exists. Thent here is the MySQL timestamp 'feature' - the first timestanp field (and only the first) when UPDATEd is updated to NOW() regardless of whether or not you include it in the updated feilds.
The only reason why they want the DB changed is because they have lazy IT departments that want to do nothing. Their IT staff does not understand the complexities of SQL DBs to them it is just an DB, and "thay're all compabible through SQL" yeah, right. "What about ODBC?" ODBC is fine for basic record operations, but the moment you try to do anyhing advanced, you're out of luck.
I used MySQL as an example, but I've worked with several other Databases, and there is no Holy Grail of DB abstraction. There's much more to databases than inserting updating, deleting and selecting rows, depsite what IT thinks.
Addendum (Score:2)
This is not my fault, this is theirs. They want to take no risks with disrupting other things, so the price of that is another server. They bring it on themselves.
My angst is directoed towards the ones who will not install our seotware on their servers, won't provide an app server, and demand that I MUST port it to their DB. It works for selling to large companies, but when you're talking hundreds or thousands of new isntallations a year, I just cannot port to every DBMS under the sun.
So your spp server problem is one that you can point the finger back at yourself for. You seem to know more than the company who wrote the software from the ground up. After all, we know nothing about databases, and we picked the wrong one from the start....
Re:As a vendor, customers are idiots (Score:2)
I would just explain the costs involved and offer to port it to their database but charge an extravagant fee for the development. After that then say use database A since this is what my program is designed to use and it will be hell of alot cheaper. But if you want to spend more money that is up to you.
Re:As a vendor, customers are idiots (Score:2)
Not to mention that if you want the application to perform well you must use vendor specific extensions.
UML? (Score:2)
As one who's worked on the vendor side... (Score:5, Insightful)
2. Often, those applications that require Superuser for the installation are third party applications. Install it yourself. You may have more trouble on the outset, but you'll know what's going on with your machine.
3. Often, vendors have their programmers as installers. The bad news is that you see the problems you do with the installations. The good news is that they'll know exactly why they need root - and they'll tell you what they need done. This might need to be a tag team installation, of course.
4. Remember, you can always invite them up there if you want to pay them for their time. Remote access is requested because it's cheaper. Alternatively, put in the contract that you want installation instructions. It will take more time, but you're always welcome to pay more.
Many of these vendor problems are reduced to cost-cutting measures. If you want to pay more, then vendors will be willing to oblige.
Re:As one who's worked on the vendor side... (Score:2)
IBM was one of the vendors I was talking about
(At least for one particular IBM product.)
Why the hell?.... (Score:2)
Ball-Peen Hammer on each of his fingers (Score:2)
They quickly agreed, since I got layered security in place (router, software, NAT, etc..), they'll have the devil's own time just making the attempt since it'll take me an half hour just to configure the network to allow for tunneled access.
Pure and simple. Oh, and forge a contract in stone saying that if you do allow access, you wash your hands of any fuckup that they perform, since it's their own dammed fault in the first place. And ah, if they do fuck it up, both contracts are broken, their boss gets a nice call from your attorneys, PLUS your own boss, etc, etc.... IE, Things MIGHT get a little ugly for the vendor in question.
IF they want your business, they will have to cooperate with you in order to maintain a good business relationship.
After all, it's a buyer's market right now.
This was solved a *long* time ago.... (Score:2)
Oh, but that's mainframes, where they run whole companies, not workstations/servers that, uh, oops... Well, maybe you could lower yourself to provide a real test environment for the vendors, and y'all from the company take responsibility for final testing and promotion to production?
mark, m'frame, *and* pc, *and*
workstation/server experience
What is the big fuss? (Score:3, Insightful)
The standard practice of firewalls to restrict access, and lower priviledge accounts are all good (and important). But the proper protection should have been negotiated into the service contract in terms of access the vendor requires and liability. Durring that negotiation process the technical authority should have considered the security concerns (and added costs). If the technical authority was inept the best you can do is minimize the risks now, and use this example to raise security concerns for the next contracts.
Zone it in Solaris10 (Score:2)
Similiar to FreeBSD Jails you can totally isolate the process but I think Solaris Zones partition a whole instance of the OS which gives it more functionality than BSD Jails. For example you can access ports lower than 1024 as root but in a seperate Solaris partition.
I am a little ignorant about this since I have just been reading about it. Anyone here know more info?
An easy solution? (Score:3, Interesting)
On a similar note, us logging RDP possible? (Score:4, Interesting)
Which makes me wonder why I can't record an RDP session somehow. This would serve two purposes: a) it would let me replay what a vendor had done later when it is more convenient and b) it would give me some proof if I later have to go blast a vendor for doing something absurdly stupid.
The same question for VNC or really any kind of remote management tool that is likely to be used on the Windows platform. Can any of them be logged and/or replayed?
- JoeShmoe
.
Re:On a similar note, us logging RDP possible? (Score:2)
These serve mainly to create demos and stuff like that.
Hope it helps.
A few words from a "third party" programmer ... (Score:2)
I develop and code a web-based database application, that is installed on a dedicated server at our client's sites. In most cases, we also install some kind of remote access software for maintainance. We access this software via Modem or ISDN with or without callback, VPN or special firewall configurations via internet. Some clients even expose the server to the internet.
Our big advantage is that we need only one TCP port (80 for http or 443 for https) with one protocol (HTTP), this is rarely a problem for any kind of firewall or network admin. For remote access, we can use different software with different requirements, depending on the wishes of the clients. SMTP, POP3 and IMAP4 can be handled very similar. Other server applications can be much harder, even such "simple" things like a Windows fileserver needs various TCP and UDP ports, and firewalling such applications can be a major PITA. Even FTP needs at least two ports and contains IP addresses inside the data packages, but either passive mode or a smart firewall can help firewalling FTP.
So, if you "just" need some software based on well-known internet standard protocols, a dedicated server behind a firewall should do the trick. If the vendor wants remote access, add a protected (i.e. callback + token) modem or ISDN line. If you need propritary software, maybe even in a non-IP environment, the things become harder. You may even need to build your own firewall or proxy, perhaps based on Linux or *BSD.
Tux2000
Another vendor view about security (Score:2)
2) I prefer be firewalled off, so I know their stuff won't mess with ours and our stuf won't mess with theirs - but the truth is most customers don't want to go thru that effort or cost. Also, unfortunately, most have no clue what their doing other than following a checklist. This becomes painfully obvious when somthing doesn't work and you need their help to diagnose and fix it.
3) The customers who have their own specialized VPN systems that require us to connect only thru that - may think they're more secure, but their not because it means that we half to use plain passwords for access from their provided box rather than out own encrypted 1024 bit key tunnel originating from an internal server.
4) At least for me, the boxes I work with have only our stuff on it. That means if things go to hell on that system it is me and my company that suffer the responsibility. So in terms of their network and other critical systems, having admin really changes very little - other than perhaps permiscous mode, and listening on used ports or ones lower than 1024.
5) It amazes me how many customers don't understand scp and ssh, or even their own VPN's.
6) One time we had a system that we didn't have access to, and it got a virus. I would have loved to take responsibility for it, but I couldn't because I can't do much to prevent that if I don't even have Admin access to the box. And also, it came from one of their internal servers - well what do you want me to do about that? Most of the time, I find that we need more protection from a customers internal network - then they need from us.
SE/Linux and XEN (Score:2)
http://sf.net/projects/selinux
http://sf.net/projects/xen
the first project allows you to grant root access to lusers, thereby convincing the program that it's got root access, but the SELinux security kicks in as well, which is far more flexible than the 20+-year-old unix security model, and most importantly SELinux doesn't give a rats arse about what a superuser is.
the second project, xen, is like vmware only faster and is a bit like where vmware installs the screen/keyboard/mouse driver and you _have_ to use those drivers. xen-linux compiles as a completely new architecture (named, duh, xen) and it comes with its own virtual device drivers.
by compartmentalising an SE/Linux XEN machine you can restrict the pants off of the vendor's software and can blow it away with ease if they start dicking about.
Try using zones.... (Score:2)
You would still want to take additional precautions if you let outsiders on your systems even in a zone. Monitor what is done and record all changes.
Re:Try using zones.... (Score:2)
What we do at work (Score:2)
Translation: (Score:2)
"Hey do you think they bought it?"
Do you have a legal department? (Score:2)
So my advice would be to go explain it to the lawyers. If it scares them (meaning if you communicate effectively why is scares you) they might have enough clout to kill the deal, causing others to try to convince the vendor to write the app in a way that is more secure for their customers.
Re:Shameless plug (Score:4, Funny)
Re:Shameless plug (Score:2)
And yes, we have remote connections available to us without a VPN.
Re:Find a new vendor (Score:5, Insightful)
I work at for a large government contractor, where root privilage is truely guarded to only those who need to know as there is data that absolutely need to be protected. Allowing random people from random companies access is not an option. It is actually a criminal offense to let someone else use your account who does not also have an account on the system (it is still a security violation even if they do have an account and is possibly subject to disciplanary action, up to and including criminal punishment depending on what it was being used for).
Basically, the only people who have root are the actual system administrators. And that is even locked down in the sense that we are supposed to log in with our regular account first and "su" to root, or otherwise need to call up security and let them know what system you need to log into directly as root (login and audit logs are checked on a regular basis for any discrepencies).
Basically, like I said, if they need root, they go thru one of us to do something.
Re:Find a new vendor (Score:3, Interesting)
Dual auditing. My activity is logged by a system I have no control over, and that other admin is logged by my system. It's true there are ways I could cover my tracks but it would be apparent I had hidden something even if they couldn't tell what I hid. At which point my ass would no longer be covered (logging has advantages for both sides in that sense), and I'd be asking if you want fries with that.
That's how my shop does it at least.
Re:Find a new vendor (Score:2)
Simple, all system administrators require the "need to know" of any and all data being stored or processed by those particular systems. It can be a pain, but it is the only way to do it. Bringing in new administrators is a time intensive process as a result of this. It also limits the people who can be system administrators as well, since foreign born, dual-citizens, or non-citizens are explicitly excluded since they can not pass the security requirements. This also means that the administrators have very intense security screening by the government as they do have access to any and all the data on those systems. For absolutely critical data, typewritters and paper are still used. This is rare where I work, but this is the proceedure if it were to occur.
Re:Find a new vendor, follow-up (Score:2)