Preventing My Hosting Provider From Rooting My Server? 539
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
If they do this.. (Score:5, Insightful)
.. just switch providers. I'm sure there are companies that treat you better.
Switch or Bail (Score:1, Insightful)
Sounds like this is in your hosting contract. Either switch, or if your that concerned, host it yourself, not in a data center. Every data center is going to say "Prove it" if you try to pin an issue on them.
remove their ssh key from the ~/.ssh directory (Score:2, Insightful)
look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.
Comment removed (Score:5, Insightful)
Stop being a douche (Score:5, Insightful)
Irony... (Score:1, Insightful)
You call yourself "hacker" but you don't already know how to do this.
Re:Stop being a douche (Score:5, Insightful)
Name and Shame (Score:4, Insightful)
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
Re:If they do this.. (Score:3, Insightful)
Indeed. Besides, why do they need the root password? How about "please give me an extract of logfiles x, y and z (if syslog doesn't do), from time hh:mm to hh:mm"? That's what they are after it seems. Or how about setting up user that has read-only access to just those log files, and give that account to CS?
Secondly, if you allow a third party direct access to your hardware, then that third party can at any time access all your data, no matter what you do software-wise. Encryption just makes it a little harder. They ARE the man in the middle if need be. A hosting provider you will have to trust to respect your privacy - if you do not have that trust you'd better not put your data in their hands. It seems in this case that trust isn't there, for whatever reason, then better move to another provider and sleep better after that.
How do they Root your Box? (Score:2, Insightful)
Re:If they do this.. (Score:5, Insightful)
As a network admin, I've run across "I know what I'm doing" people in the past. FWIW, I'm often times that guy when I'm calling tech support. It's one part ego, one big part actually knowing what I'm doing. I don't want to go through tech support 101 with some monkey on the phone when I know what the issue is.
Having said that, there have been times when I thought I knew what the issue was, but it turned out to be something else. I think that a hosting provider wanting access to log files is perfectly reasonable. They aren't arbitrarily asking for the files. The questioner states that he is having problems and he asked them to sort it out. Tech support 101 says to look at the log files. The questioner doesn't make it clear whether or not he offered to give them the log files.
Is the hosting provider a bit off base? Yes and no. Yes, it's kind of lame that they are rooting boxes. On the other hand, the questioner might be more problems than he is worth from their point of view. If I were in the same situation, I'd just change providers and find one who will put into writing that they won't root my box (good luck with that).
(Car Analogy) - It's like leasing a car with a repair warranty and wanting to do your own repairs. You diagnose the cause of the problem and take the car to the mechanic. You ask the mechanic to fix your car under warranty and he asks you for your keys. You refuse to give him the keys.
It seems to me that if a person can't fix a problem on their own, and that person then asks for help fixing the problem, they need to give up some control to the person they have asked for help from. Unless a person selects a hosting provider with an SLA that will give them physical access to their hardware on a 24/7 basis, that person is going to have to make some accomodation (like providing access to log files) when the hosting provider needs to get involved with troubleshooting.
Shutting you down to investigate your spamming (Score:3, Insightful)
Re:Stop being a douche (Score:5, Insightful)
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
I find these to be unacceptable terms.
you might be our customer (Score:5, Insightful)
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
Re:Stop being a douche (Score:5, Insightful)
You say this
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
....however, from another post you let the truth slip out
they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change (...LATER...) When they migrated it from Savvis to some datacenter in Dallas 2 months ago.....
So you openly admit the machine IS NOT YOURS. You are essentially keeping them from their own machine, which I find unethical. I can't blame them for taking matters into their own hand and rebooting the system into single-user mode and locking you out until you play nice.
Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere.
Re:Other side (Score:2, Insightful)
If they wanted to retain access after you've changed the password, they could have easy enough.
Re:You're complicating things. (Score:3, Insightful)
Your MTA melting due to incoming connections is not the fault of their network. It's your box. Fix it, or get someone else to, or don't run an MTA (srsly, SENDMAIL? The 90s called, they want their line noise configuration back). If the connections never transfer any data, maybe SYN cookies would help? (is there a full TCP handshake?) Did you get a new IP when you moved?
And $35 isn't that much to pay. Surely you're paying several times that per month for the hosting, and if not, their margins are thin enough that you can't expect them to jump through whatever hoops your paranoia requires.
Usually more to the story than this.... (Score:4, Insightful)
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
Re:If they do this.. (Score:2, Insightful)
> It seems to me that if a person can't fix a problem on their own, and that person then asks for help
> fixing the problem, they need to give up some control to the person they have asked for help from.
Close but still not quite the root of the problem here. It is a common one, a mismatch between responsibility and authority. The guy was demanding the hosting provider assume responsibility beyond the authority he was willing to give them. In the end the hosting provider claimed the matching authority to the responsibility the customer was holding them to and all hell broke loose. They should have simply closed his trouble ticket as CANTFIX when he refused them access to the information they needed to work on his problem and let him leave in a snit. A troublemaker like this customer would have been equally pissed off but the hosting provider would have gone into court (where this will almost certainly end up) with a rock solid case.
Re:Stop being a douche (Score:3, Insightful)
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Okay.. so now you admit you don't even own the DRIVE. Even better. Sorry, but my conclusion is that no matter what agreements your hosting provider may have with others, YOU are the one in the wrong here -- not them.
Have them burn the data (which you more than likely own) onto a CD/DVD, then host it yourself since you claim to be so much more competent then they are.
Re:If they do this.. (Score:5, Insightful)
I have a mixture of co-lo (ie where I own the box) and full-server rental, and the latter is treated much like the former for me. Occasionally chaos and cock-up has happened, but nothing worse.
When you the renter of space are managing a raw server then the hosting company should understand at the very least that you may be hosting private data (eg banking details) that they never want to incur vicarious liability for the misuse of, eg if the hoster were to gain unauthorised root access to your maachine and then customers of the Web site were to suffer financial losses soon after...
Rgds
Damon
Re:If they do this.. (Score:2, Insightful)
Re:If they do this.. (Score:4, Insightful)
I third this.
When our provider started having numerous unexplained outages, we quietly deployed equipment to a new provider across town and changed the DNS. I don't even think they asked us why we didn't renew our contract.
There's just no reason to do business with people like this. Leave - as fast as you can.
A.
Re:If they do this.. (Score:4, Insightful)
IANAL, but I'm pretty sure that's irrelevant. You can't be bound to terms of a contract which are illegal. If your provider cracked your root password and logged into your server, they have committed the crime of illegal trespassing upon a computer system whether it's in the contract or not.
Wrong.
If I take $5 from my wallet and put it down on my porch table, you cannot normally just take it without committing the crime of theft. However, if you and i form a contract that any money left on my porch can be taken by you, well, then that's part of the contract, not theft.
The essential part of contracting is that you exchange something you have ($) for something the other guy has (internet hosting.) Absent the contract, neither of you are entitled to what the other has; the contract is the precise manner in which you exchange those things.
If you buy hosting from someone else, KEEP A COPY of the contract, and stay abreast of any changes. If you do not understand completely every part of it, hire a lawyer to have it explained to you. (Or just ask for that part to be re-written to be clearer.)
Re:If they do this.. (Score:3, Insightful)
There are lots of really good providers out there. Enough so that if there's any little thing that you're not happy with, you ought to let your current provider know immediately, and then change.
Even the suggestion that they need root access to help you is enough that you ought to leave right away. If they don't know how wrong that is, then who knows what else they think is "standard practice"?
Just the fact that your system went down several times in one day, on more than one occasion should also be an indication that you should find a better provider.
I'm beginning to understand how your ISP feels. (Score:3, Insightful)
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
Re:You're complicating things. (Score:5, Insightful)
Re:If they do this.. (Score:3, Insightful)
Re:If they do this.. (Score:4, Insightful)
Re:If they do this.. (Score:3, Insightful)
Yeah, host in a proper data center, but supply your own hardware (ISPs that provide hardware typically buy the absolute cheapest hardware they can get hold of) and absolutely ensure that your server has some kind of lights out management support... That way you can recover from any software problem (even sofar as reinstalling the os) and should be easily able to diagnose any hardware or network related problems.
Re:How do they Root your Box? (Score:1, Insightful)
After reading all this...
Bottom line is. You are fucking stupid for STILL doing business with this company.
Some companys suck.
Some companys which didn't suck before, will in the future.
You know how to fix this already.
There is a very good reason they're doing this (Score:2, Insightful)
Use SELinux (Score:3, Insightful)
Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.
Re:If they do this.. (Score:4, Insightful)
IANAL, but I'm pretty sure that's irrelevant. You can't be bound to terms of a contract which are illegal. If your provider cracked your root password and logged into your server, they have committed the crime of illegal trespassing upon a computer system whether it's in the contract or not.
Wrong.
Sorry, but you're the one that is wrong. Your analogy sucks and is wrong. Here's an equivalent analogy, if you contract with someone, that they can have any $5 bill you leave on your dinner table inside your house, it is still illegal for them to break into your house to get it.
You cannot write a contract that permits illegal activity. knowingly writing a contract to allow criminal activity is prima facie proof of criminal conspiracy to commit said crime.
That said, he could have a contract that allows them to have access to his computer, in which case his refusal to give them access is in violation of the terms of the contract, and they may be able to disconnect him for that. They however are not allowed to commit misdemeanors and/or felonies, aka rooting a server, to get access to what is allowed them via the contract. Now if his contract says they are allowed to root his server, I'd be very surprised, but it still wouldn't hold up in a court. Really onerous terms in a contract are not enforceable, or legal. If the service provider is really doing this, I can assure you it is illegal for them to do so. If the contract says they can, then the employees doing it are at risk of prison as are the lawyers/persons who wrote the contract, and the management who are allowing it to happen.
I'd like to know what evidence the poster has that his server has been rooted. Furthermore, if his server is so easily rooted, I'd request that he stop using the internet, and remove all his machines at once. We don't need any more people contributing to the botnets. If you can't maintain your systems so they can't be rooted at the drop of a hat, then you have no business having servers on the internet.
My advice to this guy is:
1) learn how to properly maintain your system,
2) switch to a new hoster,
3) provided he has suitable proof of their unauthorized access, find the applicable law [ncsl.org] and prosecute.
Re:If they do this.. (Score:4, Insightful)
I'm not disputing that. However, rooting the server because the client doesn't want to give you the root password is a bit much, don't you think? Wouldn't it be more appropriate to say, "Sorry, no root password, not fix." and let it go at that?
Re:I'm beginning to understand how your ISP feels. (Score:1, Insightful)
I work in a data center, and sooner or later you run into guys like this. They wont allow you to run the diagnostic tests to prove/reproduce an issue, nor give any access to the server for staff to verify from that end. They go out of their way to be obstructive, and refuse to allow proper verification of any sort other than their word. Even with log files provided, there shouldn't be an issue letting them verify the data. I'm not saying provider techs are always right, but have a little faith in them. If the OP is as stubborn as he comes across, hes a customer from hell. If you dont do EXACTLY what they are demanding, they have a hissy fit.
Assuming this is a rented dedicated server, the ISP/DC owns the hardware and have every right to verify issues as they see fit before doing any form of replacement or further action to rectify the problem. Read the TOS very carefully, I know we have clauses in ours for this specific scenario so customers cant randomly demand replacements without any verification according to our guidelines.
Re:You're complicating things. (Score:3, Insightful)
"Hey landlord my heat is broken for the third time since you changed out the external heat pump unit. I think that's broken."
"Ok, can I come over today and fix it? I'll need you to leave all your safes unlocked and open, and you cannot be present while I'm there."
"Nope, never, sorry. Im giving you my notice and suing you for no heat"..
FTFY.
Re:How do they Root your Box? (Score:2, Insightful)
I am sorry but without the additional details that have been requested a few times this thread is going no where fast. I would advise that /. drop this thread unless additional info is provided.
1. what type of hosting contract. ( Own or Rent server )?
2. Dedicated server or shared?
3. Link to hosting company Terms of Service?
This "hacker" guy. (Score:1, Insightful)
This "hacker" guy is actually causing a bit of a stir on the Drobo forums, accusing support left-right-and-centre of destroying his data. Only a couple of days before he started screaming bloody murder he was posting questions about "tuning" his filesystems with tune2fs.
Shame the Drobo forums are for customers only, but he's a bit of a tit. I wouldn't believe a single word he says about the ISP.
Re:If they do this.. (Score:3, Insightful)
Sorry, but you're the one that is wrong. Your analogy sucks and is wrong. Here's an equivalent analogy, if you contract with someone, that they can have any $5 bill you leave on your dinner table inside your house, it is still illegal for them to break into your house to get it.
If you add a term that allows them to break in, why not? (Of course such a contract wouldn't normally exist in the real world).
If you lose the keys to your house and hire a locksmith or whoever to crack your doors open, then he's breaking into your house, legally.
You cannot write a contract that permits illegal activity. knowingly writing a contract to allow criminal activity is prima facie proof of criminal conspiracy to commit said crime.
For a lot of crimes which "harms" another, consent is a defense. It is not a crime for you to use my computer if I consented to that. And a contract is good evidence of consent.
If you hire security professionals to poke at your systems to find possible exploits, are you committing a conspiracy to hack your own systems? I think not.
Re:If they do this.. (Score:4, Insightful)
If you buy hosting from someone else, KEEP A COPY of the contract, and stay abreast of any changes. If you do not understand completely every part of it, hire a lawyer to have it explained to you. (Or just ask for that part to be re-written to be clearer.)
With the number of contracts people make daily, one would go broke due to consultation fees before he gets anything done.
Besides, reading the comments of people who apparently have an opinion on how the law is, I think the danger is more in the terms where you *think* you understand what it says.
Colo is your answer (Score:1, Insightful)
Based on your previous replies, you have said that the server is not yours.
It really doesn't matter WHO the server belongs to after that, it simply is not yours.
Whether they rent it, re-sell it or whatever, it still is not yours.
Honestly, I don't care how "clued" you are, they are not wrong in asking for the root password to diagnose a problem which you claim is happening with their hardware. ( I say "their" since they are in a contract with someone else over this server and you are not in that contract). If you feel they are that inept, you should have kept detailed notes and asked to speak to management to voice concern about their previous ineptness to see if a more senior technician can work on the issue.
Keep in mind that a good business would at least want to try to see if there is a problem with the machine in question so that they can they replace it with those they are renting from. At the company where I previously worked, they rented their machines for a period of time and that worked out better than buying new machines every few years. If anything went seriously wrong during that period, it was a matter of shipping the machine back and getting a new one at the same rental fee.
Now, as to them locking you out and all that, I'd have to see YOUR contract with them to know what is right and wrong regardless of how inept you think they are.
If your contract allows this behavior, then you really have no room to complain.
If they hardware was determined to be the issue, who knows if they had the exact hardware to stuck you back on (since they rent the hardware). Its not clear and I honestly do not feel like reading through more replies here.
It sounds like you made things harder on yourself than needed. But you chose to pay the 35/day KVM switch and fixed things yourself (good for you). BUT, that was YOUR CHOICE in not giving up your password.
I also question WHY they would try to hide their tracks on rooting your box as they did. If its in their contract, so what? Hiding it makes it suspicious.
At any rate, the short version is what I said in the title.
You need to get a machine and colo it. Get the necessary equipment as has been previously stated and at that point, you have legal recourse. As it stands, I don't know what re-course you have as that depends on your contract with them.
Example: As a company I worked for, NOT providing the requested information and/or logs was reason enough to close an open trouble ticket. We normally gave our best effort since some situations existed where people genuinely could not do so (security clearances, etc). Once we hit a point where that info was non-optional, they customer had a choice to make and that was do what they had to in order to get the logs or close the ticket.
Now is the time for you to continue to make your choices.
* Abide by their rules and fess up the password (pursuing through management as needed)
* pay KVM charges as needed to avoid giving the password
* change providers that might more suit your whims (good luck on that)
* COLO