Preventing My Hosting Provider From Rooting My Server? 539
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?"
Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
If they do this.. (Score:5, Insightful)
.. just switch providers. I'm sure there are companies that treat you better.
Re:If they do this.. (Score:5, Informative)
Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.
Re: (Score:3, Insightful)
Re:If they do this.. (Score:4, Insightful)
Re: (Score:3, Interesting)
The upload is mediocre at about 1 mbit up. Makes for slow transfers over VPN. Synchronous services are still far too expensive here.
Re: (Score:3, Insightful)
Yeah, host in a proper data center, but supply your own hardware (ISPs that provide hardware typically buy the absolute cheapest hardware they can get hold of) and absolutely ensure that your server has some kind of lights out management support... That way you can recover from any software problem (even sofar as reinstalling the os) and should be easily able to diagnose any hardware or network related problems.
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
It was not the fastest but I was only paying about ~$150/mo and that was with 8 IP's.
Today, unless you live in the sticks (my brother, who ironically works for Comcast but is too far out for their service, does and even he has 1Mbit down DSL - recently acquired) getting a 1 Mbit or better up sounds doable for not too awful much via DSL (maybe I a
Re: (Score:3, Insightful)
There are lots of really good providers out there. Enough so that if there's any little thing that you're not happy with, you ought to let your current provider know immediately, and then change.
Even the suggestion that they need root access to help you is enough that you ought to leave right away. If they don't know how wrong that is, then who knows what else they think is "standard practice"?
Just the fact that your system went down several times in one day, on more than one occasion should also be an in
Re:If they do this.. (Score:4, Interesting)
Even the suggestion that they need root access to help you is enough that you ought to leave right away
You've not dealt with many *nix users fubaring their configuration settings and then moaning about the hardware being bad have you?
Re:If they do this.. (Score:5, Informative)
I _DO_ work at a hosting provider, and unfortunately root access is often required to repair the steaming piles of crap customers often leave behind.
That may be a symptom of the type of customer we attract, but I don't think this is unusual. The submitter is an exception, most people who get them have no business operating a server.
For the submitter: get an internet KVM and use LUKS to encrypt. You'll need the KVM to remotely type your passphrase. They can still get at it if they really wanted to - but you aren't going to be worth the effort.
Hell if you are where I think you are, you better check your boot scripts, I think you'll find openvt opening a terminal where you may not expect.
Re:If they do this.. (Score:4, Insightful)
I'm not disputing that. However, rooting the server because the client doesn't want to give you the root password is a bit much, don't you think? Wouldn't it be more appropriate to say, "Sorry, no root password, not fix." and let it go at that?
Re: (Score:3, Interesting)
As I said to someone else, I'm not talking bout colo. We own all the hardware in question, the customer "rents" it. In any case they are perfectly within their rights to tell us not to touch, and we will obey. However, by default we will access your server as needed without your explicit permission.
This policy is in place because 90% of our customers will send in a ticket "My site is down! Fix!" and half the time they don't even give us an IP or domain, let alone access credentials.
Re:If they do this.. (Score:4, Informative)
How about a padlock on the box, and a BIOS password?
Unfortunately they sound like the type of people who would cut the lock, and reset the BIOS. I think the poster should find a new colo and tell us who the current colo is so we can avoid them.
Re: (Score:2)
I agree. This provider sounds very fishy, if they are intentionally breaking into your hardware without your permission. Get another provider, post haste, that has a privacy clause in their contract guaranteeing that they will not do such a thing without your explicit permission, and that if they do something outside the contract such as breaking into your box without your permission, there's a rather steep monetary fee to pay on their part as a breach of contract lawsuit is in order.
Re:If they do this.. (Score:4, Informative)
I also agree.
No need for a provider to do this to you at all.
I use three different providers covering different parts of the world and none of them would dream of doing anything like that.
On the other hand if I *ask* them to help rescue me, they are happy to.
Rgds
Damon
Re:If they do this.. (Score:5, Informative)
I definitely agree. The local staff at my colos are happy to do simple tasks while acting as my eyes and performing keyboard instructions on my behalf (if it's critical) or even simply exchanging a dvdr in a backup burner, otherwise they need to (and would) stay away. But those are my boxes in a rack and any network outages could be confirmed by the datacenter's logging and equipment.
I get the impression that OP doesn't have his own equipment in a rented rack, otherwise hardware would be solely on OP's shoulders. If you are using their equipment, I don't feel that it's unreasonable to ask you for logs to diagnose, however they should have gone about it legitimately with you sharing it to them.
Screw this paranoia about encryption, The Man isn't gonna come after your FOSS site and it just adds additional complexity that needs to be troubleshooted when things go south. If your sites are so heavily trafficked, buy your own box to eliminate one of the things you are blaming on the provider and move over to a provider who will not fuck with your box on a whim and respects you.
Re:If they do this.. (Score:5, Insightful)
I have a mixture of co-lo (ie where I own the box) and full-server rental, and the latter is treated much like the former for me. Occasionally chaos and cock-up has happened, but nothing worse.
When you the renter of space are managing a raw server then the hosting company should understand at the very least that you may be hosting private data (eg banking details) that they never want to incur vicarious liability for the misuse of, eg if the hoster were to gain unauthorised root access to your maachine and then customers of the Web site were to suffer financial losses soon after...
Rgds
Damon
Re:If they do this.. (Score:5, Informative)
Bogons, UK
GetNetworks/JavaServletHosting, US
WebVisions, AsiaPac (currently India and Australia)
Rgds
Damon
Re:If they do this.. (Score:4, Interesting)
Have them charged with illegally accessing your machine. Add in a claim for damages for the costs and time that is necessary to get the computer up and running again.
It may be a little harsh, but your Attorney General cannot refuse to prosecute this, as it would set a precedent. Any refusal to prosecute, would allow for a lawsuit of selective enforcement of the law.
You'll probably have your ISP booting you as a customer, but it sounds like you don't really want them anyway.
Comment removed (Score:5, Insightful)
Re: (Score:3, Interesting)
Definitely.
First, do your homework, make sure you didn't accidentally give them consent in your TOS with them.
Second, if you didn't give that consent, contact a lawyer (for civil litigation), and then notify authorities.
Whatever you do, don't tolerate it.
Re: (Score:2)
I'll second this. If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony. You may just want to notify the FBI, as well as your and their state's attorneys general.
Re:If they do this.. (Score:4, Interesting)
If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony.
Hmmm ... I wonder how many ISPs have carefully worded their TOS "agreement" so that a passage that sounds innocent (or meaningless) to the typical legal "layman" actually says that they have your permission to access any equipment plugged into their lines. I can see and ISP, especially one with a local monopoly, deciding that they can probably get away with doing this to their customers.
Do we actually have to hire a lawyer to go over such "agreements" to verify that we haven't signed away all rights to them in exchange for service? Or are there likely to be laws that would classify such terms as unconscionable? And since IANAL, how would I recognize such terms hidden out in the legalese?
Note that we have had a number of stories in recent years that were based on a clause in an ISP's TOS doc saying that anything you put on their machines was legally their property. Remember when msn.com used this defense when they were caught extracting images of customers' kids from their email and web sites and using them in advertising? There have been a number of warnings to musicians that putting your music on a "personal web site" that's on an ISP's machine may constitute assigning your copyright to the ISP, as could emailing your own creations via an email server that belongs to your ISP. So some ISPs do have a history of making legal claims on their customers' property, often basing the claim on TOS phrases that most people without legal training wouldn't understand.
Re:If they do this.. (Score:4, Insightful)
IANAL, but I'm pretty sure that's irrelevant. You can't be bound to terms of a contract which are illegal. If your provider cracked your root password and logged into your server, they have committed the crime of illegal trespassing upon a computer system whether it's in the contract or not.
Wrong.
If I take $5 from my wallet and put it down on my porch table, you cannot normally just take it without committing the crime of theft. However, if you and i form a contract that any money left on my porch can be taken by you, well, then that's part of the contract, not theft.
The essential part of contracting is that you exchange something you have ($) for something the other guy has (internet hosting.) Absent the contract, neither of you are entitled to what the other has; the contract is the precise manner in which you exchange those things.
If you buy hosting from someone else, KEEP A COPY of the contract, and stay abreast of any changes. If you do not understand completely every part of it, hire a lawyer to have it explained to you. (Or just ask for that part to be re-written to be clearer.)
Re:If they do this.. (Score:4, Insightful)
IANAL, but I'm pretty sure that's irrelevant. You can't be bound to terms of a contract which are illegal. If your provider cracked your root password and logged into your server, they have committed the crime of illegal trespassing upon a computer system whether it's in the contract or not.
Wrong.
Sorry, but you're the one that is wrong. Your analogy sucks and is wrong. Here's an equivalent analogy, if you contract with someone, that they can have any $5 bill you leave on your dinner table inside your house, it is still illegal for them to break into your house to get it.
You cannot write a contract that permits illegal activity. knowingly writing a contract to allow criminal activity is prima facie proof of criminal conspiracy to commit said crime.
That said, he could have a contract that allows them to have access to his computer, in which case his refusal to give them access is in violation of the terms of the contract, and they may be able to disconnect him for that. They however are not allowed to commit misdemeanors and/or felonies, aka rooting a server, to get access to what is allowed them via the contract. Now if his contract says they are allowed to root his server, I'd be very surprised, but it still wouldn't hold up in a court. Really onerous terms in a contract are not enforceable, or legal. If the service provider is really doing this, I can assure you it is illegal for them to do so. If the contract says they can, then the employees doing it are at risk of prison as are the lawyers/persons who wrote the contract, and the management who are allowing it to happen.
I'd like to know what evidence the poster has that his server has been rooted. Furthermore, if his server is so easily rooted, I'd request that he stop using the internet, and remove all his machines at once. We don't need any more people contributing to the botnets. If you can't maintain your systems so they can't be rooted at the drop of a hat, then you have no business having servers on the internet.
My advice to this guy is:
1) learn how to properly maintain your system,
2) switch to a new hoster,
3) provided he has suitable proof of their unauthorized access, find the applicable law [ncsl.org] and prosecute.
Re: (Score:3, Insightful)
Sorry, but you're the one that is wrong. Your analogy sucks and is wrong. Here's an equivalent analogy, if you contract with someone, that they can have any $5 bill you leave on your dinner table inside your house, it is still illegal for them to break into your house to get it.
If you add a term that allows them to break in, why not? (Of course such a contract wouldn't normally exist in the real world).
If you lose the keys to your house and hire a locksmith or whoever to crack your doors open, then he's breaking into your house, legally.
You cannot write a contract that permits illegal activity. knowingly writing a contract to allow criminal activity is prima facie proof of criminal conspiracy to commit said crime.
For a lot of crimes which "harms" another, consent is a defense. It is not a crime for you to use my computer if I consented to that. And a contract is good evidence of consent.
If you hire security professionals to poke at your systems to find poss
Re:If they do this.. (Score:4, Insightful)
If you buy hosting from someone else, KEEP A COPY of the contract, and stay abreast of any changes. If you do not understand completely every part of it, hire a lawyer to have it explained to you. (Or just ask for that part to be re-written to be clearer.)
With the number of contracts people make daily, one would go broke due to consultation fees before he gets anything done.
Besides, reading the comments of people who apparently have an opinion on how the law is, I think the danger is more in the terms where you *think* you understand what it says.
Re: (Score:3, Informative)
You are right and wrong. An example of something I can't do is give you permission via contract to kill me. I can't do this even via a power of attorney where you are acting on my behalf since suicide is illegal. In this case, the crime does not depend on my consent.
But in any case where the action is only a crime without my consent, the contract constitutes the consent. Breaking and Entering is only breaking and entering if you don't have a legal right to access the property/home for instance, that right c
Re: (Score:2)
No fucking kidding. I'd be looking for the door post haste if anyone in their tech team asked for my root password.
Re:If they do this.. (Score:5, Interesting)
I might ask for more evidence that the provider actually rooted the server before pronouncing judgment. I'm not saying that the person posing the question is lying, but simply because I don't have enough evidence either way.
Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true.
That said, I don't know either way really. It could be argued one way or another. If I were a provider, I might even insist upon the ability to access systems running on my network simply because of liability concerns as the provider. I as the provider can't be allowing untoward activity on my network.
That all said, and without actually proclaiming judgment one way or another, in the end if you're not happy with your provider for any reason, whether reasonable or not, you should just leave them and find a new one.
Re:If they do this.. (Score:5, Insightful)
As a network admin, I've run across "I know what I'm doing" people in the past. FWIW, I'm often times that guy when I'm calling tech support. It's one part ego, one big part actually knowing what I'm doing. I don't want to go through tech support 101 with some monkey on the phone when I know what the issue is.
Having said that, there have been times when I thought I knew what the issue was, but it turned out to be something else. I think that a hosting provider wanting access to log files is perfectly reasonable. They aren't arbitrarily asking for the files. The questioner states that he is having problems and he asked them to sort it out. Tech support 101 says to look at the log files. The questioner doesn't make it clear whether or not he offered to give them the log files.
Is the hosting provider a bit off base? Yes and no. Yes, it's kind of lame that they are rooting boxes. On the other hand, the questioner might be more problems than he is worth from their point of view. If I were in the same situation, I'd just change providers and find one who will put into writing that they won't root my box (good luck with that).
(Car Analogy) - It's like leasing a car with a repair warranty and wanting to do your own repairs. You diagnose the cause of the problem and take the car to the mechanic. You ask the mechanic to fix your car under warranty and he asks you for your keys. You refuse to give him the keys.
It seems to me that if a person can't fix a problem on their own, and that person then asks for help fixing the problem, they need to give up some control to the person they have asked for help from. Unless a person selects a hosting provider with an SLA that will give them physical access to their hardware on a 24/7 basis, that person is going to have to make some accomodation (like providing access to log files) when the hosting provider needs to get involved with troubleshooting.
Re:If they do this.. (Score:5, Informative)
If your hosting provider wants the log files, they don't need root, just a copy of the files. Give them a user-level login, and put a copy of the files where that user can see them.
The outage already happened, right? They don't need the current logs as they happen, just the logs for the outage period.
Re: (Score:3, Informative)
Syslog (and it's variants) already provides the functionality so a provider does not have to access a server. I can't think of a reason a provider needs to access a server other than to test their ability to sniff passwords. Hopefully the OP is exchanging ssh keys with their server.
Granted that, in this case, the provider wan
Re: (Score:3, Insightful)
Indeed. Besides, why do they need the root password? How about "please give me an extract of logfiles x, y and z (if syslog doesn't do), from time hh:mm to hh:mm"? That's what they are after it seems. Or how about setting up user that has read-only access to just those log files, and give that account to CS?
Secondly, if you allow a third party direct access to your hardware, then that third party can at any time access all your data, no matter what you do software-wise. Encryption just makes it a little ha
Re: (Score:2)
Linode has been excellent and they never mess with my stuff.
Re: (Score:2)
You should switch, but not because a better provider won't root your servers, but because you might not have to submit support tickets if their side of the network doesn't have problems.
Every hosting provider has Terms Of Use. They have every right to go into your system, and just because you encrypt everything or deny access, it doesn't mean they won't flat out unplug your service. In fact, the best providers are better because they are good at preventing high loads due to violations. They prevent them by
Re:If they do this.. (Score:5, Informative)
I used to lease a dedicated box, and over the years, I was faced with this decision to switch to another provider on 4 separate occasions. A similar situation, they weren't always asking for the root password, but in each instance, there were hardware problems crashing the box, and they would play ring around the rosies fixing it, and my family's business was losing business and credibility. I understand the problem, for $200/mo. for a dedicated box, a company can't afford to have a gaggle of techs so they can provide 4 hour response time, and have hot spare boxes ready to roll into place.
We decided we could no longer employ "hosting provider roulette" as part of a reasonable business plan.
I found a data center not exactly close to home but within a reasonable distance, near Downtown L.A., that had a reasonable colocation rate. We put together a 1U box, and put it in the rack. For $125/mo (~$40/mo. less than we were paying for an inferior dedicated box) our down time has all but disappeared. The thing is, whenever the down time was because of the hardware, I was able to drive down there and swap stuff around, including swapping in a tower for a time while I had to send our server out for repair. Our down time profile changed from multi-week periods of unreliable service to brief windows of usually less than an hour though one time about 4 hours while I had to drive around town rounding up some new drives once.
Another thing we got out of this move was the ability to configure our box as we pleased. We upgraded out box to an 8 core box with 24GB of RAM and a 1.3TB RAID 10 array. Leasing a box like that is cost prohibitive. And the time to do this was minimal, I just ordered the parts from Newegg, built it, burned it in, and went down to perform the swap. They didn't quibble about me having two machines hooked up for a day while I made the swap.
The "company" that runs the data center is actually a few companies sharing a space, and they help each other out covering tech support at night. They are all 100% top-notch geeks, who understand the problems a web admin faces, and they are very accommodating. They will put an IP KVM on the box or even wheel up a head, plug it in, and tell you what the screen is saying, even help diagnose, all for no additional charge. You can hire them to be a monkey by the hour, if needed, or just go there 24x7x365 on a moment's notice, to access the data center, which is secured, has halon, backup chillers, redundant power and backbone feeds, UPS, diesel generator, etc. all the amenities. I get nothing from them except goodwill for my recommendation. I can tell you I have never once in the 6 years I have colocated a box with time, have I ever considered moving. For anything. Not even the cloud could beckon me away. If anyone is interested: http://colocation.la/ [colocation.la] also http://serverlogistics.com/ [serverlogistics.com] if you are interested in shared or dedicated hosting.
Re:If they do this.. (Score:4, Insightful)
I third this.
When our provider started having numerous unexplained outages, we quietly deployed equipment to a new provider across town and changed the DNS. I don't even think they asked us why we didn't renew our contract.
There's just no reason to do business with people like this. Leave - as fast as you can.
A.
Just.. (Score:5, Funny)
Use chmod (Score:3, Informative)
This is very simple (Score:5, Interesting)
1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.
2. Figure out how they broke in. If they broke in then someone else likely could too.
I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.
Re: (Score:2)
It sounds like they "broke in" by booting from alternate media and reading the hard disk. They have physical access to the hardware--there's not a lot you can do to stop them.
Why don't you have any remote management? (Score:2)
Re:Why don't you have any remote management? (Score:5, Informative)
I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.
One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.
I would assume any reasonable host would be willing to get you a similar sort of hookup.
remove their ssh key from the ~/.ssh directory (Score:2, Insightful)
look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.
Illegal? (Score:5, Informative)
Depending on where the center is located [ncsl.org], and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.
I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.
Find a company that wants to keep your business. (Score:2)
Change ISPs. My colo company specifically states in our contract that they will not touch my server (physically or remotely) without my prior consent. Once they had to rearrange the rack to fit a new server in, and they called me to ask if I wanted to be present or to move mine myself.
Other side (Score:5, Interesting)
On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.
They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."
Re: (Score:2)
Re: (Score:2)
When we were in situations like that, before we got our own datacenter and colocated, we would always do the same thing. We'd make sure everything was backed up (for if something got screwed up).
Then we'd change the root password, and give them the new one. That way they could look around at whatever they needed, change what they needed.
When they said they were done, we'd change the root password back. They had all the access they needed, but couldn't mess with stuff the rest of the time.
Re: (Score:3, Informative)
Stop being a douche (Score:5, Insightful)
Re:Stop being a douche (Score:5, Insightful)
Re: (Score:2)
From TFC:
From TFS:
Re:Stop being a douche (Score:5, Insightful)
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
I find these to be unacceptable terms.
Re:Stop being a douche (Score:5, Insightful)
You say this
I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.
....however, from another post you let the truth slip out
they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change (...LATER...) When they migrated it from Savvis to some datacenter in Dallas 2 months ago.....
So you openly admit the machine IS NOT YOURS. You are essentially keeping them from their own machine, which I find unethical. I can't blame them for taking matters into their own hand and rebooting the system into single-user mode and locking you out until you play nice.
Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere.
Re:Stop being a douche (Score:4, Interesting)
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Basically I am leasing a physical server from company (A) who is leasing it from company (B), and that too may not be the end of the line. (B) may not own it either, and they may be colocating hardware from company (C) or (D) somewhere in there.
So whose TOS am I subject to here? Who is violating whose laws? It gets curiouser and curiouser the more I dig into it.
Re: (Score:3, Insightful)
Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.
Okay.. so now you admit you don't even own the DRIVE. Even better. Sorry, but my conclusion is that no matter what agreements your hosting provider may have with others, YOU are the one in the wrong here -- not them.
Have them burn the data (which you more than likely own) onto a CD/DVD, then host it yourself since you claim to be so much more competent then they are.
You're complicating things. (Score:5, Interesting)
You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.
I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?
Re: (Score:2)
At the end of the day though, someone somewhere at the hosting company is still going to be able to reboot your server into a rescue environment and reset the root password. Go colocation if you're really that paranoid about it.
A good encrypted filesystem setup can be sorted such that nothing can be mounted without your external influence. At this point the host will not be able to get hold of the data from a rescue environment as the keys are external to the server. Of course this means that in a genuine reboot situation (such as a power outage that lasts longer than the UPS can survive) you will have to intervene (providing the keys) to start the services again which will be a hassle if it happens at a bad moment like the middle of the night if you have no support cover who have access to the keys too.
Re:You're complicating things. (Score:4, Informative)
They offer KVM access, at $35.00/day, which in this case I refuse to pay to fix what they broke, outside of the context of the server. They migrated me from one chassis to another with completely different hardware, causing my machine to go offline. They want me to pay $35.00 for 24-hours of KVM access to reconfigure the network to support the hardware they moved things to.
Alternately, they want me to hand over the root password (not a privileged account, but THE root password), so they can do it themselves. Since I installed, configured and manage the OS entirely on this machine, and they've demonstrated their ineptitude before, I'm not giving them root. Ever.
Because this server has been running 24x7 for about 3 years without a single outstanding issue. When they migrated it from Savvis to some datacenter in Dallas 2 months ago, I've had no less than 20 separate outages , while the underlying OS and application stack itself has not changed in any way to facilitate those outages.
In every single case, they demand that I give them the root password, so they can diagnose the issues on the machine. In every single case, I've shown them nagios, ntop, hotsanic, sar, etc. logs demonstrating that the OS itself is not the cause of the outages.
For example, since this migration to Dallas, every other Sunday between 7:00am and 8:00am EST, my server's load goes over 100 as incoming connections spike over 700/sec., sendmail refuses connections due to the load, and the box seizes up. The logs show that the connections are established and then hang. NOTHING on the machine triggers every other Sunday between these hours that would cause that.
Only a few days ago, they indicated that the NIC on the server may be causing the issues. I'm down 2-3 hours every other Sunday because of this.
They're not asking for the logs, they're asking for root. That's a completely separate (and unacceptable) solution to their own problems outside of the box itself.
Re: (Score:3, Insightful)
Your MTA melting due to incoming connections is not the fault of their network. It's your box. Fix it, or get someone else to, or don't run an MTA (srsly, SENDMAIL? The 90s called, they want their line noise configuration back). If the connections never transfer any data, maybe SYN cookies would help? (is there a full TCP handshake?) Did you get a new IP when you moved?
And $35 isn't that much to pay. Surely you're paying several times that per month for the hosting, and if not, their margins are thin
Re: (Score:3, Informative)
Re:You're complicating things. (Score:5, Insightful)
Re:You're complicating things. (Score:4, Informative)
Yes, they "rent" a KVM to customers for $35.00/USD for a 24-hour period, unfortunately...
In this case, to break the standoff between myself and the hosting provider, I yielded and had them invoice me for the $35 so I could get the server up, rip the data off of it, terminate my services with them and go after them for financial compensation for the damages, downtime (12 day outage 2 months ago without an apology), etc.
Re: (Score:3, Insightful)
"Hey landlord my heat is broken for the third time since you changed out the external heat pump unit. I think that's broken."
"Ok, can I come over today and fix it? I'll need you to leave all your safes unlocked and open, and you cannot be present while I'm there."
"Nope, never, sorry. Im giving you my notice and suing you for no heat"..
FTFY.
Dell Drac (Score:2, Interesting)
Face palm (Score:2)
Among the many choices you have, you can install a remote monitoring/administration card.
But that's really using technology to solve the wrong problem. The problem is your ISP.
Fire your ISP. You already have two very good reasons for doing so. First, they
should simply ask for the logs, not demand entry into the system. Second, for taking
down your server, breaking into it (what if you had data on there you didn't want
unauthorized people to see?) without your express, positive, verified consent.
Using technolo
rooted? What does this word mean? (Score:2)
"...They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs.."
What does the word "rooted" mean in this case? They did not have the root password so what is the posted referring to here?
Disclaimer: I am no computer geek.
Re: (Score:2)
Re: (Score:2)
It means that they bypassed the root user password. This is easily accomplished when you have physical access to the machine. It's often used for recovering from a forgotten root password.
http://www.felipecruz.com/blog_reset-root-password-linux.php [felipecruz.com]
Re: (Score:2)
The Planet (Score:2)
My NDA has long since expired - I'm open to answering questions via email if anyone has them.
Re: (Score:3, Informative)
Name and Shame (Score:4, Insightful)
If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.
As others have pointed out
They have physical access, so game over (Score:2)
Sue them. And switch to a different company.
More details please? (Score:5, Informative)
However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
Hardware problem? (Score:2)
Why did you not just give them access to the logfiles. Just setup a new apache on port 8080, do a few symlinks to bring the log files into the default html folder, update config to follow symlinks, add a .htaccess file and you are done. Should not take more then 20 minutes.
How exactly did you expect them to help you, if you are the only customer with problems, and you don't give them any access to your log files.
And it's time to change hosting provider if they really did the "bringing it down and poking aro
You can do this (Score:5, Informative)
My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.
If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.
First, go elsewhere. (Score:2)
First: switch providers. Do not put up with this behaviour.
The only thing you can do otherwise it use encrypted filesystems for your data (you don't need to encrypt *everything* including the root filesystem, just main data store(s) like /home & any databases & sensitive logs stored elsewhere and temporary storage areas) without storing any trace of the keys on the server or anywhere else accessible by the server. Have the server request (or otherwise wait for) the keys to be provided by you before
Tough question... (Score:5, Funny)
"How do I turn a whore into a housewife?"
Some things are only solved by replacment.
So why is it crashing? (Score:5, Informative)
The logs should tell you why the machine crashed.
How busy was the server?
There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).
Yubikey and YubiPAM (Score:2)
Simple:
1. Require all passwords to be Yubikey OTP passwords on any login prompt.
2. Refuse access, and only give them the logs manually.
3. When they shut your server down and open it up to yank the drive, hit 'em with a breach of lawsuit.
4. ????
5. PROFIT!
Re: (Score:2)
Some solutions... (Score:2)
How about:
a) when they ask for root, change it to something innocuous, let them log in and do their stuff, then change it back.
b) change hosting providers.
c) host it yourself. Get a business class internet connection to your office/home/etc, rig up a closet with power and A/C. And if you need "five nines", get a second power provider and a UPS, and a second internet connection.
Ultimately, physical access is everything.
And yeah, why are you asking for help if you're not going to let them help you? Are you
Use Xen and run your apps in virtual machines (Score:2)
Get a dedicated server running the latest centos or ubuntu server release. Use Xen to run your various applications in dedicated virtual machines. You can encrypt entire domains in a number of ways both internal and external. A dedicated test domain can be set up for your hosting provider to access, etc.
How do they Root your Box? (Score:2, Insightful)
Re:How do they Root your Box? (Score:5, Informative)
They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.
Shutting you down to investigate your spamming (Score:3, Insightful)
It sounds like... (Score:2)
It sounds like you have a "Managed Server" type of plan with your hosting provider. With a managed plan, a provider has some legal obligations (despite customer instructions) to maintain the host. Go find an "Unmanaged" hosting provider, or colo your own equipment.
you might be our customer (Score:5, Insightful)
Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:
1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.
2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.
You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.
The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.
Re: (Score:3, Informative)
This IS an unmanaged plan. All the provide is ping and power, I do the rest. I manage the OS, the configuration and everything else. This is not VPS, I lease a physical server, and they don't touch it.
I had the same situation.. (Score:4, Informative)
I'm beginning to understand how your ISP feels. (Score:3, Insightful)
They're also denying me KVM access, unless I pay $35.00 for it, so I can go in and fix the networking they changed when they moved my drive to a completely different chassis without my knowledge or approval.
Since you are not disclosing the ISP name so we can examine their TOS or contracts to see who's really being the jerk here and learn enough actually help you, pay the $35/day just to recover/delete your data if you need to and find another host that suits you.
Otherwise STFU; I'm beginning to understand how your ISP feels.
Usually more to the story than this.... (Score:4, Insightful)
First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.
There's basically three plans we have.
#1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.
#2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.
#3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.
So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go
possible way they do it (Score:3, Informative)
Use SELinux (Score:3, Insightful)
Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.
Re: (Score:2)
Only problem I see with that is that if he swapped the BIOS I'm pretty sure that any hardware warranty support would basically go away... I'm assuming he bought "supported hardware" through a vendor though (such as Dell or HP).... if the servers are "off brand" this might work
Any company of a decent size who doesn't want to go through the hassle of supporting their hardware end-to-end will usually go with a vendor though (Google's an exception... they have the resources to support the servers they hack toge
Re: (Score:2)
That would prevent them from just browsing. It wouldn't prevent them from hijacking the binaries used to mount the encrypted disk.
So really, it all depends upon how hard they want to work to get into your box.