Preventing My Hosting Provider From Rooting My Server? 539
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
This is very simple (Score:5, Interesting)
1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.
2. Figure out how they broke in. If they broke in then someone else likely could too.
I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.
Re:If they do this.. (Score:4, Interesting)
Have them charged with illegally accessing your machine. Add in a claim for damages for the costs and time that is necessary to get the computer up and running again.
It may be a little harsh, but your Attorney General cannot refuse to prosecute this, as it would set a precedent. Any refusal to prosecute, would allow for a lawsuit of selective enforcement of the law.
You'll probably have your ISP booting you as a customer, but it sounds like you don't really want them anyway.
Other side (Score:5, Interesting)
On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.
They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."
Re:If they do this.. (Score:5, Interesting)
I might ask for more evidence that the provider actually rooted the server before pronouncing judgment. I'm not saying that the person posing the question is lying, but simply because I don't have enough evidence either way.
Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true.
That said, I don't know either way really. It could be argued one way or another. If I were a provider, I might even insist upon the ability to access systems running on my network simply because of liability concerns as the provider. I as the provider can't be allowing untoward activity on my network.
That all said, and without actually proclaiming judgment one way or another, in the end if you're not happy with your provider for any reason, whether reasonable or not, you should just leave them and find a new one.
You're complicating things. (Score:5, Interesting)
You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.
I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?
Dell Drac (Score:2, Interesting)
Re:Illegal? (Score:1, Interesting)
A buddy of mine hosted VOIP servers for years. Due to widespread licensing violations Ventrilo switched to a new licensing system; you were given individualized software which "called back home" to note how many servers a provider was actually hosting to ensure they paid the licensing fee. Someone at the data center in conjunction with an illegal hosting company accessed one of their servers and copied the software. So this illegal host was hosting tons of servers without paying a license fee, my buddy was on the hook for all of the extra servers.
While they knew what happened at the data center as it was later found out they've done similar deeds before they couldn't prove anything. The host was sued and lost but the data center got off scot free.
What I'm trying to say is that it is hard to prove things when they can simply lie and further even if you can prove it there isn't much you can do.
Re:Other side (Score:1, Interesting)
"The guy then refuses access when they try to figure out what's going on so that they can fix it."
But the "it" is not the server. This is like you complaining there's a huge pothole on the road, and the DOT demanding the keys to your vehicle. (DOT then proceeds to investigate said problem by using your vehicle to drive them to lunch, having lunch in your car in your driveway while checking out your wife working in your yard, and determining said pothole exists by ramming your vehicle over it repeatedly at high speed.)
And while not explicitly stated, you also overlook that they, after rooting said server, don't seem to have solved the problem anyways, providing further evidence that this has nothing to do with the server.
This has become so common these days; people don't do their own due diligence, and instead blame the "complainer" for being unreasonable. Sounds like this guy has the Comcast of hosting.
Re:If they do this.. (Score:3, Interesting)
Definitely.
First, do your homework, make sure you didn't accidentally give them consent in your TOS with them.
Second, if you didn't give that consent, contact a lawyer (for civil litigation), and then notify authorities.
Whatever you do, don't tolerate it.
Re:Stop being a douche (Score:1, Interesting)
It is very simple.. you are operating on their terms as long as you are having your equipment hosted on their side. I hate to be blunt, but host it yourself if you don't want to give others access to your system for troubleshooting.
Re:Stop being a douche (Score:4, Interesting)
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Basically I am leasing a physical server from company (A) who is leasing it from company (B), and that too may not be the end of the line. (B) may not own it either, and they may be colocating hardware from company (C) or (D) somewhere in there.
So whose TOS am I subject to here? Who is violating whose laws? It gets curiouser and curiouser the more I dig into it.
Re:If they do this.. (Score:4, Interesting)
Even the suggestion that they need root access to help you is enough that you ought to leave right away
You've not dealt with many *nix users fubaring their configuration settings and then moaning about the hardware being bad have you?
Re:If they do this.. (Score:3, Interesting)
The upload is mediocre at about 1 mbit up. Makes for slow transfers over VPN. Synchronous services are still far too expensive here.
Re:If they do this.. (Score:1, Interesting)
The parent is absolutely correct in that you can't be bound to illegal terms in a contract. In your example, the contract contains no illegal terms. Theft requires a lack of consent, and thus, by contracting consent, there is no theft.
Think of it more in these terms. If you and I contract to kill someone -- even one of the parties to the contract* -- you can't sue in court to enforce the terms, since it is illegal. Likewise with pretty much any crime involving a third party, since an unknowing party can't give consent. Other illegal acts don't fly, even with consent. E.g., if I sign a contract to buy bulk cocaine from you, good luck enforcing it.
In this case, even if the contract between the remote admin and the hosting provider grants the provider the admin's consent for root access the server, it does NOT automatically grant the host the right to crack the password. No court would find that reasonable, just like me asking someone to stop by my house and pick something up and then smashing a window to get in because the key didn't work in the lock.
*Great story I read one time about an obese man who had serious health problems and needed to lose weight. He actually took a hit contract out on himself, which was to be enforced if he neglected to go to the gym each day. According to the former hitman, of course, so I can't guarantee the reliability to the story.
Re:If they do this.. (Score:2, Interesting)
The poster sent a "WTF" ticket to the provider. The provider at that point was ASKED to become involved in troubleshooting. If you know what is going on and where the problem exists then state what the issue is and then the provider can fix it. If you ask me WTF?!?!?! I would ask you for root access on your box too depending on the problem. I want to see not only just logs but everything. Did you make some weird change to the kernel? Did you modify a lib file? If you don't want me to look into the problem don't ask for my help.
The questioner has an attitude of I know what I am doing and you have a problem so fix it. The provider does have a problem they want to fix it but also investigate. If it is hardware they want to troubleshoot it properly and replace only failed components not the entire system. If it is the network they need to find out is the problem with the router, switch, network cable, or network card.
I think for those of us that know a lot of what we are doing in technology we tend to except someone to just take our word for it. However, coming from the background of working in Call Centers I know also that the other guy doesn't know that I know what I am doing and he still has to check to make sure it is right. How many times have we all heard the customer go on and on about how the problem is our fault and we find out that the customer didn't turn the computer on or forgot to plug the mouse in?
My point being if you don't want my help please don't ask for it. If you want my help then please give me the ability to help. Offering logs would have been fine... If you are doing something so secretive that you don't want them to see something then just move companies to one you can trust enough to let them see it. Offer to have one of the techs or some of the techs sign and agree to a NDA and return to you.
Re:Stop being a douche (Score:2, Interesting)
I call bullshit on your story.
If they wanted your root password that bad, they'd have it, you obviously aren't booting from an encrypted drive so they could just single user boot your machine and do whatever they wanted, since you claim they are rebooting it.
There is no logical reason for them to lock you out of your own machine to get you to give them a root password.
There is more to this story that you're leaving out, intentionally I suspect.
Perhaps I'm wrong but my guess is you should just pay them or stop trying to scam them, whichever it is and stop giving slashdot some bullshit line of crap in hope that you'll get someone else to give you a sneaky way to turn the tables on them.
Re:If they do this.. (Score:3, Interesting)
It was not the fastest but I was only paying about ~$150/mo and that was with 8 IP's.
Today, unless you live in the sticks (my brother, who ironically works for Comcast but is too far out for their service, does and even he has 1Mbit down DSL - recently acquired) getting a 1 Mbit or better up sounds doable for not too awful much via DSL (maybe I am living under a rock?). I could host mail and most other things I need to do across that (I did on a 384k line). If I were trying to do a heavy usage VPN or web services that gets tons of hits per day that would be a problem of course. It almost sounds like the OP is more concerned about security than speed (I am stabbing in the dark a little here) - in which case, taking the box home is a great idea. I loved it when my services here right at home. Now that I am on Comcast I only have 1 IP.... You can only host so many services (1 per port) off that connection.
Re:If they do this.. (Score:4, Interesting)
If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony.
Hmmm ... I wonder how many ISPs have carefully worded their TOS "agreement" so that a passage that sounds innocent (or meaningless) to the typical legal "layman" actually says that they have your permission to access any equipment plugged into their lines. I can see and ISP, especially one with a local monopoly, deciding that they can probably get away with doing this to their customers.
Do we actually have to hire a lawyer to go over such "agreements" to verify that we haven't signed away all rights to them in exchange for service? Or are there likely to be laws that would classify such terms as unconscionable? And since IANAL, how would I recognize such terms hidden out in the legalese?
Note that we have had a number of stories in recent years that were based on a clause in an ISP's TOS doc saying that anything you put on their machines was legally their property. Remember when msn.com used this defense when they were caught extracting images of customers' kids from their email and web sites and using them in advertising? There have been a number of warnings to musicians that putting your music on a "personal web site" that's on an ISP's machine may constitute assigning your copyright to the ISP, as could emailing your own creations via an email server that belongs to your ISP. So some ISPs do have a history of making legal claims on their customers' property, often basing the claim on TOS phrases that most people without legal training wouldn't understand.
Encrypt your root partition (Score:1, Interesting)
I've almost finished writing a HOWTO for full disk encryption on a Linode.
It'll be on their wiki in a few days time.
Re:If they do this.. (Score:3, Interesting)
Re:If they do this.. (Score:3, Interesting)
As I said to someone else, I'm not talking bout colo. We own all the hardware in question, the customer "rents" it. In any case they are perfectly within their rights to tell us not to touch, and we will obey. However, by default we will access your server as needed without your explicit permission.
This policy is in place because 90% of our customers will send in a ticket "My site is down! Fix!" and half the time they don't even give us an IP or domain, let alone access credentials.