Preventing My Hosting Provider From Rooting My Server? 539
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
don't trust em' (Score:1, Informative)
XEN FTMFW.
http://www.howtoforge.com/creating-a-fully-encrypted-para-virtualized-xen-guest-system-using-debian-lenny
Re:If they do this.. (Score:5, Informative)
Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.
Use chmod (Score:3, Informative)
Re:If they do this.. (Score:4, Informative)
I also agree.
No need for a provider to do this to you at all.
I use three different providers covering different parts of the world and none of them would dream of doing anything like that.
On the other hand if I *ask* them to help rescue me, they are happy to.
Rgds
Damon
Illegal? (Score:5, Informative)
Depending on where the center is located [ncsl.org], and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
Re:Why don't you have any remote management? (Score:5, Informative)
I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.
One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.
I would assume any reasonable host would be willing to get you a similar sort of hookup.
More details please? (Score:5, Informative)
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
You can do this (Score:5, Informative)
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
Re:If they do this.. (Score:5, Informative)
Bogons, UK
GetNetworks/JavaServletHosting, US
WebVisions, AsiaPac (currently India and Australia)
Rgds
Damon
So why is it crashing? (Score:5, Informative)
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
Re:How do they Root your Box? (Score:5, Informative)
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
Re:You're complicating things. (Score:4, Informative)
They offer KVM access, at $35.00/day, which in this case I refuse to pay to fix what they broke, outside of the context of the server. They migrated me from one chassis to another with completely different hardware, causing my machine to go offline. They want me to pay $35.00 for 24-hours of KVM access to reconfigure the network to support the hardware they moved things to.
Alternately, they want me to hand over the root password (not a privileged account, but THE root password), so they can do it themselves. Since I installed, configured and manage the OS entirely on this machine, and they've demonstrated their ineptitude before, I'm not giving them root. Ever.
Because this server has been running 24x7 for about 3 years without a single outstanding issue. When they migrated it from Savvis to some datacenter in Dallas 2 months ago, I've had no less than 20 separate outages , while the underlying OS and application stack itself has not changed in any way to facilitate those outages.
In every single case, they demand that I give them the root password, so they can diagnose the issues on the machine. In every single case, I've shown them nagios, ntop, hotsanic, sar, etc. logs demonstrating that the OS itself is not the cause of the outages.
For example, since this migration to Dallas, every other Sunday between 7:00am and 8:00am EST, my server's load goes over 100 as incoming connections spike over 700/sec., sendmail refuses connections due to the load, and the box seizes up. The logs show that the connections are established and then hang. NOTHING on the machine triggers every other Sunday between these hours that would cause that.
Only a few days ago, they indicated that the NIC on the server may be causing the issues. I'm down 2-3 hours every other Sunday because of this.
They're not asking for the logs, they're asking for root. That's a completely separate (and unacceptable) solution to their own problems outside of the box itself.
I had the same situation.. (Score:4, Informative)
Re:If they do this.. (Score:5, Informative)
I used to lease a dedicated box, and over the years, I was faced with this decision to switch to another provider on 4 separate occasions. A similar situation, they weren't always asking for the root password, but in each instance, there were hardware problems crashing the box, and they would play ring around the rosies fixing it, and my family's business was losing business and credibility. I understand the problem, for $200/mo. for a dedicated box, a company can't afford to have a gaggle of techs so they can provide 4 hour response time, and have hot spare boxes ready to roll into place.
We decided we could no longer employ "hosting provider roulette" as part of a reasonable business plan.
I found a data center not exactly close to home but within a reasonable distance, near Downtown L.A., that had a reasonable colocation rate. We put together a 1U box, and put it in the rack. For $125/mo (~$40/mo. less than we were paying for an inferior dedicated box) our down time has all but disappeared. The thing is, whenever the down time was because of the hardware, I was able to drive down there and swap stuff around, including swapping in a tower for a time while I had to send our server out for repair. Our down time profile changed from multi-week periods of unreliable service to brief windows of usually less than an hour though one time about 4 hours while I had to drive around town rounding up some new drives once.
Another thing we got out of this move was the ability to configure our box as we pleased. We upgraded out box to an 8 core box with 24GB of RAM and a 1.3TB RAID 10 array. Leasing a box like that is cost prohibitive. And the time to do this was minimal, I just ordered the parts from Newegg, built it, burned it in, and went down to perform the swap. They didn't quibble about me having two machines hooked up for a day while I made the swap.
The "company" that runs the data center is actually a few companies sharing a space, and they help each other out covering tech support at night. They are all 100% top-notch geeks, who understand the problems a web admin faces, and they are very accommodating. They will put an IP KVM on the box or even wheel up a head, plug it in, and tell you what the screen is saying, even help diagnose, all for no additional charge. You can hire them to be a monkey by the hour, if needed, or just go there 24x7x365 on a moment's notice, to access the data center, which is secured, has halon, backup chillers, redundant power and backbone feeds, UPS, diesel generator, etc. all the amenities. I get nothing from them except goodwill for my recommendation. I can tell you I have never once in the 6 years I have colocated a box with time, have I ever considered moving. For anything. Not even the cloud could beckon me away. If anyone is interested: http://colocation.la/ [colocation.la] also http://serverlogistics.com/ [serverlogistics.com] if you are interested in shared or dedicated hosting.
Re:If they do this.. (Score:5, Informative)
I definitely agree. The local staff at my colos are happy to do simple tasks while acting as my eyes and performing keyboard instructions on my behalf (if it's critical) or even simply exchanging a dvdr in a backup burner, otherwise they need to (and would) stay away. But those are my boxes in a rack and any network outages could be confirmed by the datacenter's logging and equipment.
I get the impression that OP doesn't have his own equipment in a rented rack, otherwise hardware would be solely on OP's shoulders. If you are using their equipment, I don't feel that it's unreasonable to ask you for logs to diagnose, however they should have gone about it legitimately with you sharing it to them.
Screw this paranoia about encryption, The Man isn't gonna come after your FOSS site and it just adds additional complexity that needs to be troubleshooted when things go south. If your sites are so heavily trafficked, buy your own box to eliminate one of the things you are blaming on the provider and move over to a provider who will not fuck with your box on a whim and respects you.
Re:I had the same situation.. (Score:2, Informative)
How am I expected to change the root password to let them in, when they've denied me access to the server unless I hand over the current root password? They're not asking for logs, they're demanding the root password; those are two very-different issues entirely.
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Re:you might be our customer (Score:3, Informative)
This IS an unmanaged plan. All the provide is ping and power, I do the rest. I manage the OS, the configuration and everything else. This is not VPS, I lease a physical server, and they don't touch it.
colo or STFU (Score:1, Informative)
The OP sounds like one of the thousands of self-important pricks I've spoken to in the 6 years I've spent in hosting. Nobody gives a shit about your server, or your projects. They're just doing their job, and faced with a "fuck you you're not getting my password" I've personally reset passwords to 50+ character passwords once I'm done.
Don't like it? Either build your own damn datacenter, or find a provider to sell you power, ping, and pipe on the 97 and manage a server you built yourself. If you own the machine, you can do whatever asinine, paranoid, double-secret encryption scheme you want.
Of course, if the machine is going down "mysteriously" and you need these "tech monkeys" to look at your logs, I highly doubt you're enough of an admin to handle coloing your own servers.
Re:If they do this.. (Score:5, Informative)
If your hosting provider wants the log files, they don't need root, just a copy of the files. Give them a user-level login, and put a copy of the files where that user can see them.
The outage already happened, right? They don't need the current logs as they happen, just the logs for the outage period.
Re:If they do this.. (Score:2, Informative)
Agreed. The host should respect your privacy and never access the data without your consent. You should switch.
If you need to give access in the future you could setup a user account, load a screen, sudo bash in there, and have them 'screen -x' so you can see what they do. Or you can tar up the logs and send them a copy.
And if you want privacy I would strongly urge you to use disk encryption to keep them out of your files. And rebuild your kernel without USB, Firewire, and PCMCIA support. There's ways to compromise this, but at least it raises the bar.
Re:you might be our customer (Score:2, Informative)
Re:Other side (Score:3, Informative)
Re:You're complicating things. (Score:3, Informative)
Re:Name and Shame - LayeredTech (Score:2, Informative)
It's obvioius David's provider for gnu-designs.com is Layered Tech. In my opinion he'd be WAY better off going to another provider; Layered Tech hosts spammers, malware purveyors and all sorts of net scum. We have LT firewalled for quite a while now. In the past they never respond to abuse complaints so we got tired of their crap and just completely blocked them. Move on to someone else, even AT&T would be preferable to LayeredTech.
possible way they do it (Score:3, Informative)
Re:If they do this.. (Score:2, Informative)
Thirded. I work for an ISP, if I tried the stunt of rooting a customer's box after the customer explicitly said "no", I'd be out and in the welfare queue in minutes. No ISP needs to directly view your logs to determine and fix errors. I know what my network is doing and I have my own logs to show it. All I need to do is show my netowrk is working per the contract, and bill the customer for traffic used. What's on the box is the customer's business, what flows through our network from the box is our business.
This is all assuming that the customer doesn't have a contract where I look after the server for them. In that case, it's our hardware and we get paid to admin the box and keep things running. And that contract is clearly labelled as such, even it's name leaves you in no doubt that the ISP has an admin account.
Re:I'm beginning to understand how your ISP feels. (Score:2, Informative)
Otherwise STFU; I'm beginning to understand how your ISP feels.
I know, I've been 2/3rds down this thread, and there are tons of helpful posts. Hacker here just keeps responding with the same shit over and over and over again.
Look, Hacker, you fucked up by not moving providers after the first incident. You come across as a total jackass here, and probably also to your provider. If the server is worth $35 to you, then pay the $35 and fix the damned thing, then move providers. If not, then start up a new account somewhere else and restore it from a backup. (If you don't have backups, that's also your fault.)
So suck up, swallow your goddamned pride, stop being so paranoid, and deal with the goddamned problem. Period.
Guess what? You're going to get screwed sometimes in life. COPE WITH IT AND MOVE ON.
Re:If they do this.. (Score:3, Informative)
You are right and wrong. An example of something I can't do is give you permission via contract to kill me. I can't do this even via a power of attorney where you are acting on my behalf since suicide is illegal. In this case, the crime does not depend on my consent.
But in any case where the action is only a crime without my consent, the contract constitutes the consent. Breaking and Entering is only breaking and entering if you don't have a legal right to access the property/home for instance, that right can be conveyed via contract. The same is true of accessing a computer system. You can sign a contract that grants someone permission to access your computer.
All in all a simple rule of thumb is to ask if you yourself can do the thing legally. If so, you can generally give someone else permission to the do the thing via contract. A notable exception would be a power/permission you yourself acquired via non transferable contract.
Re:You're complicating things. (Score:4, Informative)
Yes, they "rent" a KVM to customers for $35.00/USD for a 24-hour period, unfortunately...
In this case, to break the standoff between myself and the hosting provider, I yielded and had them invoice me for the $35 so I could get the server up, rip the data off of it, terminate my services with them and go after them for financial compensation for the damages, downtime (12 day outage 2 months ago without an apology), etc.
Re:If they do this.. (Score:2, Informative)
Indeed I work for a hosting company and although it isn't frequent if a user reports random outages my standard response will be "Look at the server logs, or if you'd like me to do so please supply some login details".
Too many people don't know what they're looking for so offering to do if for them. I assume that if they don't trust me (as admin) they'll be hosting elsewhere and I'd always suggest they change their password(s) afterward.
Re:Some solutions... (Score:1, Informative)
5 nines is INSANELY expensive. Not even Microsoft or the large providers go past 3 nines for the most part.
If you want 5 nines, you arn't talking anout a PC in a colo, you are talking clusters in geographically separate data centers connected to SANs with multiple interfaces. You are talking virtualization with multiple floating hosts. This is costly, and even with this, there may be downtime from other unexpected sources.
Re:If they do this.. (Score:4, Informative)
How about a padlock on the box, and a BIOS password?
Unfortunately they sound like the type of people who would cut the lock, and reset the BIOS. I think the poster should find a new colo and tell us who the current colo is so we can avoid them.
Re:If they do this.. (Score:3, Informative)
Syslog (and it's variants) already provides the functionality so a provider does not have to access a server. I can't think of a reason a provider needs to access a server other than to test their ability to sniff passwords. Hopefully the OP is exchanging ssh keys with their server.
Granted that, in this case, the provider wants access to the logs to determine the cause of an outage that has already occurred isn't easier just to tee the future logs off to a syslog server of the providers choosing? I am *fairly* certain that *most* applications can log via syslog and that the output can be stream edited for sensitive information and removed allowing the server owner ultimate control of what information is shared.
I'm not saying I approve of the provider's unauthorised access to the server, I don't, but access to the system logs can be provided without said provider even logging into the system. It's a compromise that has to be negotiated because maintaining the uptime of the server is in everybody's interest.
Re:The Planet (Score:3, Informative)
Either way, the relevance to the conversation was that we were told to root a customer's box if they had a hardware complaint and wouldn't give us the root password to make sure it wasn't the software, which resulted in quite a few customers getting emails from Frank Castle and forfeiting their fees and server lease. It's just bad business, in my opinion, and it's why I left The Planet after 6 months.
Re:If they do this.. (Score:5, Informative)
I _DO_ work at a hosting provider, and unfortunately root access is often required to repair the steaming piles of crap customers often leave behind.
That may be a symptom of the type of customer we attract, but I don't think this is unusual. The submitter is an exception, most people who get them have no business operating a server.
For the submitter: get an internet KVM and use LUKS to encrypt. You'll need the KVM to remotely type your passphrase. They can still get at it if they really wanted to - but you aren't going to be worth the effort.
Hell if you are where I think you are, you better check your boot scripts, I think you'll find openvt opening a terminal where you may not expect.