Forgot your password?
typodupeerror
Security

Preventing My Hosting Provider From Rooting My Server? 539

Posted by Soulskill
from the booby-traps dept.
hacker writes "I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has 'unexpected' outages, which I've determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it's not the OS itself. When I file 'WTF?'-style support tickets to the provider through their web-based ticketing system, I often get the response of: 'Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.' Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they've done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?" Read on for a few more details of hacker's situation.
"With sufficient memory and CPU, I could install VMware and run my entire system within a VM, and encrypt that. I could also use UML, and try to bury my data in there, but that's not encrypted. Ultimately, I'd like to have an encrypted system end-to-end, but if I do that, I can't reboot it remotely without entering the password at boot time. Since I'll be remote, that's a blocker for me.

What does the Slashdot community have for ideas in this regard? What other technologies and options are at my disposal to try here (beyond litigation and jumping providers, both of which are on the short horizon ahead)."
This discussion has been archived. No new comments can be posted.

Preventing My Hosting Provider From Rooting My Server?

Comments Filter:
  • If they do this.. (Score:5, Insightful)

    by sopssa (1498795) * <sopssa@email.com> on Saturday December 26, 2009 @01:25PM (#30556750) Journal

    .. just switch providers. I'm sure there are companies that treat you better.

    • Re:If they do this.. (Score:5, Informative)

      by drinkypoo (153816) <martin.espinoza@gmail.com> on Saturday December 26, 2009 @01:31PM (#30556790) Homepage Journal

      Second this. Isn't it an adage that someone who has access to the hardware has already won? Secure some solid evidence and publicize it on your way off the host.

      • Re: (Score:3, Insightful)

        by Yvanhoe (564877)
        Or better yet : host it at your own house. Obviously the poster has enough skills to administer a NIX box. Put it at your home with a decent DSL connection and let it run. Access to the hardware is hard to beat. Even if the data are ciphered, you won't manage to deny access to the OS from the provider.
        • by flappinbooger (574405) on Saturday December 26, 2009 @05:14PM (#30558400) Homepage
          That's a great idea, except for the TOS of the DSL and the horrid upload speeds even good DSL typically has.
          • Re: (Score:3, Interesting)

            by danomac (1032160)
            I have DSL and I'm allowed to host services, even smtp. It's actually intended for businesses, but hosting a server is not an issue.

            The upload is mediocre at about 1 mbit up. Makes for slow transfers over VPN. Synchronous services are still far too expensive here.
          • Re: (Score:3, Insightful)

            by Bert64 (520050)

            Yeah, host in a proper data center, but supply your own hardware (ISPs that provide hardware typically buy the absolute cheapest hardware they can get hold of) and absolutely ensure that your server has some kind of lights out management support... That way you can recover from any software problem (even sofar as reinstalling the os) and should be easily able to diagnose any hardware or network related problems.

          • Re: (Score:3, Interesting)

            by aztracker1 (702135)
            Business DSL acounts aren't much more and allow for hosting. Many DSL providers even support ML-PPP for combining lines.
    • by erpbridge (64037)

      I agree. This provider sounds very fishy, if they are intentionally breaking into your hardware without your permission. Get another provider, post haste, that has a privacy clause in their contract guaranteeing that they will not do such a thing without your explicit permission, and that if they do something outside the contract such as breaking into your box without your permission, there's a rather steep monetary fee to pay on their part as a breach of contract lawsuit is in order.

    • Re:If they do this.. (Score:4, Informative)

      by DamonHD (794830) <d@hd.org> on Saturday December 26, 2009 @01:35PM (#30556824) Homepage

      I also agree.

      No need for a provider to do this to you at all.

      I use three different providers covering different parts of the world and none of them would dream of doing anything like that.

      On the other hand if I *ask* them to help rescue me, they are happy to.

      Rgds

      Damon

      • Re:If they do this.. (Score:5, Informative)

        by socsoc (1116769) on Saturday December 26, 2009 @03:21PM (#30557558)

        I definitely agree. The local staff at my colos are happy to do simple tasks while acting as my eyes and performing keyboard instructions on my behalf (if it's critical) or even simply exchanging a dvdr in a backup burner, otherwise they need to (and would) stay away. But those are my boxes in a rack and any network outages could be confirmed by the datacenter's logging and equipment.

        I get the impression that OP doesn't have his own equipment in a rented rack, otherwise hardware would be solely on OP's shoulders. If you are using their equipment, I don't feel that it's unreasonable to ask you for logs to diagnose, however they should have gone about it legitimately with you sharing it to them.

        Screw this paranoia about encryption, The Man isn't gonna come after your FOSS site and it just adds additional complexity that needs to be troubleshooted when things go south. If your sites are so heavily trafficked, buy your own box to eliminate one of the things you are blaming on the provider and move over to a provider who will not fuck with your box on a whim and respects you.

        • by DamonHD (794830) <d@hd.org> on Saturday December 26, 2009 @03:44PM (#30557724) Homepage

          I have a mixture of co-lo (ie where I own the box) and full-server rental, and the latter is treated much like the former for me. Occasionally chaos and cock-up has happened, but nothing worse.

          When you the renter of space are managing a raw server then the hosting company should understand at the very least that you may be hosting private data (eg banking details) that they never want to incur vicarious liability for the misuse of, eg if the hoster were to gain unauthorised root access to your maachine and then customers of the Web site were to suffer financial losses soon after...

          Rgds

          Damon

    • Re:If they do this.. (Score:4, Interesting)

      by Anonymous Coward on Saturday December 26, 2009 @01:36PM (#30556830)

      Have them charged with illegally accessing your machine. Add in a claim for damages for the costs and time that is necessary to get the computer up and running again.

      It may be a little harsh, but your Attorney General cannot refuse to prosecute this, as it would set a precedent. Any refusal to prosecute, would allow for a lawsuit of selective enforcement of the law.

      You'll probably have your ISP booting you as a customer, but it sounds like you don't really want them anyway.

      • by jcr (53032) <jcr@NoSPaM.mac.com> on Saturday December 26, 2009 @01:39PM (#30556854) Journal

        First, check your contract and make double sure that you didn't give them permission for this, and if not, go ahead and file charges.

        -jcr

        • Re: (Score:3, Interesting)

          by johnkzin (917611)

          Definitely.

          First, do your homework, make sure you didn't accidentally give them consent in your TOS with them.
          Second, if you didn't give that consent, contact a lawyer (for civil litigation), and then notify authorities.

          Whatever you do, don't tolerate it.

        • by iphayd (170761)

          I'll second this. If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony. You may just want to notify the FBI, as well as your and their state's attorneys general.

          • Re:If they do this.. (Score:4, Interesting)

            by jc42 (318812) on Saturday December 26, 2009 @10:31PM (#30560200) Homepage Journal

            If you didn't agree to them having root access in the contract, they are illegally accessing your hardware, which is a felony.

            Hmmm ... I wonder how many ISPs have carefully worded their TOS "agreement" so that a passage that sounds innocent (or meaningless) to the typical legal "layman" actually says that they have your permission to access any equipment plugged into their lines. I can see and ISP, especially one with a local monopoly, deciding that they can probably get away with doing this to their customers.

            Do we actually have to hire a lawyer to go over such "agreements" to verify that we haven't signed away all rights to them in exchange for service? Or are there likely to be laws that would classify such terms as unconscionable? And since IANAL, how would I recognize such terms hidden out in the legalese?

            Note that we have had a number of stories in recent years that were based on a clause in an ISP's TOS doc saying that anything you put on their machines was legally their property. Remember when msn.com used this defense when they were caught extracting images of customers' kids from their email and web sites and using them in advertising? There have been a number of warnings to musicians that putting your music on a "personal web site" that's on an ISP's machine may constitute assigning your copyright to the ISP, as could emailing your own creations via an email server that belongs to your ISP. So some ISPs do have a history of making legal claims on their customers' property, often basing the claim on TOS phrases that most people without legal training wouldn't understand.

    • No fucking kidding. I'd be looking for the door post haste if anyone in their tech team asked for my root password.

    • Re:If they do this.. (Score:5, Interesting)

      by JeffSh (71237) <jeffslashdot&m0m0,org> on Saturday December 26, 2009 @01:41PM (#30556880)

      I might ask for more evidence that the provider actually rooted the server before pronouncing judgment. I'm not saying that the person posing the question is lying, but simply because I don't have enough evidence either way.

      Highly intelligent people tend towards a sometimes unreasonable paranoia and sometimes make conclusions (i.e. my server was rooted to look at the logs) that are not exactly true.

      That said, I don't know either way really. It could be argued one way or another. If I were a provider, I might even insist upon the ability to access systems running on my network simply because of liability concerns as the provider. I as the provider can't be allowing untoward activity on my network.

      That all said, and without actually proclaiming judgment one way or another, in the end if you're not happy with your provider for any reason, whether reasonable or not, you should just leave them and find a new one.

      • by dave562 (969951) on Saturday December 26, 2009 @02:24PM (#30557160) Journal

        As a network admin, I've run across "I know what I'm doing" people in the past. FWIW, I'm often times that guy when I'm calling tech support. It's one part ego, one big part actually knowing what I'm doing. I don't want to go through tech support 101 with some monkey on the phone when I know what the issue is.

        Having said that, there have been times when I thought I knew what the issue was, but it turned out to be something else. I think that a hosting provider wanting access to log files is perfectly reasonable. They aren't arbitrarily asking for the files. The questioner states that he is having problems and he asked them to sort it out. Tech support 101 says to look at the log files. The questioner doesn't make it clear whether or not he offered to give them the log files.

        Is the hosting provider a bit off base? Yes and no. Yes, it's kind of lame that they are rooting boxes. On the other hand, the questioner might be more problems than he is worth from their point of view. If I were in the same situation, I'd just change providers and find one who will put into writing that they won't root my box (good luck with that).

        (Car Analogy) - It's like leasing a car with a repair warranty and wanting to do your own repairs. You diagnose the cause of the problem and take the car to the mechanic. You ask the mechanic to fix your car under warranty and he asks you for your keys. You refuse to give him the keys.

        It seems to me that if a person can't fix a problem on their own, and that person then asks for help fixing the problem, they need to give up some control to the person they have asked for help from. Unless a person selects a hosting provider with an SLA that will give them physical access to their hardware on a 24/7 basis, that person is going to have to make some accomodation (like providing access to log files) when the hosting provider needs to get involved with troubleshooting.

        • Re:If they do this.. (Score:5, Informative)

          by wytcld (179112) on Saturday December 26, 2009 @03:31PM (#30557636) Homepage

          If your hosting provider wants the log files, they don't need root, just a copy of the files. Give them a user-level login, and put a copy of the files where that user can see them.

          The outage already happened, right? They don't need the current logs as they happen, just the logs for the outage period.

          • Re: (Score:3, Informative)

            by MrKaos (858439)

            If your hosting provider wants the log files, they don't need root, just a copy of the files. Give them a user-level login, and put a copy of the files where that user can see them.

            Syslog (and it's variants) already provides the functionality so a provider does not have to access a server. I can't think of a reason a provider needs to access a server other than to test their ability to sniff passwords. Hopefully the OP is exchanging ssh keys with their server.

            Granted that, in this case, the provider wan

    • Re: (Score:3, Insightful)

      by wvmarle (1070040)

      Indeed. Besides, why do they need the root password? How about "please give me an extract of logfiles x, y and z (if syslog doesn't do), from time hh:mm to hh:mm"? That's what they are after it seems. Or how about setting up user that has read-only access to just those log files, and give that account to CS?

      Secondly, if you allow a third party direct access to your hardware, then that third party can at any time access all your data, no matter what you do software-wise. Encryption just makes it a little ha

    • Switch providers.

      Linode has been excellent and they never mess with my stuff.
    • You should switch, but not because a better provider won't root your servers, but because you might not have to submit support tickets if their side of the network doesn't have problems.

      Every hosting provider has Terms Of Use. They have every right to go into your system, and just because you encrypt everything or deny access, it doesn't mean they won't flat out unplug your service. In fact, the best providers are better because they are good at preventing high loads due to violations. They prevent them by

    • Re:If they do this.. (Score:5, Informative)

      by coolgeek (140561) on Saturday December 26, 2009 @03:19PM (#30557538) Homepage

      I used to lease a dedicated box, and over the years, I was faced with this decision to switch to another provider on 4 separate occasions. A similar situation, they weren't always asking for the root password, but in each instance, there were hardware problems crashing the box, and they would play ring around the rosies fixing it, and my family's business was losing business and credibility. I understand the problem, for $200/mo. for a dedicated box, a company can't afford to have a gaggle of techs so they can provide 4 hour response time, and have hot spare boxes ready to roll into place.

      We decided we could no longer employ "hosting provider roulette" as part of a reasonable business plan.

      I found a data center not exactly close to home but within a reasonable distance, near Downtown L.A., that had a reasonable colocation rate. We put together a 1U box, and put it in the rack. For $125/mo (~$40/mo. less than we were paying for an inferior dedicated box) our down time has all but disappeared. The thing is, whenever the down time was because of the hardware, I was able to drive down there and swap stuff around, including swapping in a tower for a time while I had to send our server out for repair. Our down time profile changed from multi-week periods of unreliable service to brief windows of usually less than an hour though one time about 4 hours while I had to drive around town rounding up some new drives once.

      Another thing we got out of this move was the ability to configure our box as we pleased. We upgraded out box to an 8 core box with 24GB of RAM and a 1.3TB RAID 10 array. Leasing a box like that is cost prohibitive. And the time to do this was minimal, I just ordered the parts from Newegg, built it, burned it in, and went down to perform the swap. They didn't quibble about me having two machines hooked up for a day while I made the swap.

      The "company" that runs the data center is actually a few companies sharing a space, and they help each other out covering tech support at night. They are all 100% top-notch geeks, who understand the problems a web admin faces, and they are very accommodating. They will put an IP KVM on the box or even wheel up a head, plug it in, and tell you what the screen is saying, even help diagnose, all for no additional charge. You can hire them to be a monkey by the hour, if needed, or just go there 24x7x365 on a moment's notice, to access the data center, which is secured, has halon, backup chillers, redundant power and backbone feeds, UPS, diesel generator, etc. all the amenities. I get nothing from them except goodwill for my recommendation. I can tell you I have never once in the 6 years I have colocated a box with time, have I ever considered moving. For anything. Not even the cloud could beckon me away. If anyone is interested: http://colocation.la/ [colocation.la] also http://serverlogistics.com/ [serverlogistics.com] if you are interested in shared or dedicated hosting.

    • by Alrescha (50745) on Saturday December 26, 2009 @03:50PM (#30557774)

      I third this.

      When our provider started having numerous unexplained outages, we quietly deployed equipment to a new provider across town and changed the DNS. I don't even think they asked us why we didn't renew our contract.

      There's just no reason to do business with people like this. Leave - as fast as you can.

      A.

  • Just.. (Score:5, Funny)

    by roblarky (1103715) on Saturday December 26, 2009 @01:29PM (#30556774)
    Be sure to stun them as soon as they start casting it.
  • Use chmod (Score:3, Informative)

    by ctrl-alt-canc (977108) on Saturday December 26, 2009 @01:34PM (#30556812)
    chmod 744 /var/log (modify the directory name as needed so that it points to where your logs reside) and they will be able to look at your logs without root password. If this is not enough for them, remember that internet is full of service provider that are eager to host you for the same money (if not less)...
  • This is very simple (Score:5, Interesting)

    by rgigger (637061) on Saturday December 26, 2009 @01:35PM (#30556816)

    1. Don't EVER host with them again. I don't know what's in your contract but as far as I understand it, breaking into your server without your permission is illegal. It's possible that you could take legal action against them.

    2. Figure out how they broke in. If they broke in then someone else likely could too.

    I have never heard of anything like that happening with any host ever. I am amazed that a company could act like that and still expect to have any customers. It's not like there aren't options.

    • by Sancho (17056)

      It sounds like they "broke in" by booting from alternate media and reading the hard disk. They have physical access to the hardware--there's not a lot you can do to stop them.

  • The only reason you wouldn't be able to remotely enter in a boot-time decryption password, is if you don't have any remote management [wikipedia.org] capabilities [wikipedia.org] on this server. If this is the case, you should get just better hardware.
    • by ottothecow (600101) <ottothecow@NosPam.gmail.com> on Saturday December 26, 2009 @01:48PM (#30556926) Homepage
      Agreed.

      I don't have too much experience in this arena but once I was running a few units and got a rack mounted sun box to play with. Thing didn't have video IIRC and it was all done via suns various terminal connections. Once I got the box set up on the rack (in a room I didnt have normal access to), I ran the terminal cable to a linux webserver that I ran on the same rack.

      One day, the sun stopped responding over its ethernet connection I thought I was screwed until I remembered that cable...sshed into the other box, brought up the terminal cable and I was soon at sun's management console that let me figure out what was going on.

      I would assume any reasonable host would be willing to get you a similar sort of hookup.

  • by Anonymous Coward

    look for a pre-authorized ssh key in ~/.ssh/authorized_keys or something similar, remove it.

  • Illegal? (Score:5, Informative)

    by DoofusOfDeath (636671) on Saturday December 26, 2009 @01:39PM (#30556856)

    Depending on where the center is located [ncsl.org], and exactly what you agreed to in your terms of service, they may have violated anti-hacking laws.

    I'm guessing that you probably won't find a district attorney who's willing to prosecute them on your behalf. But if you're outside the U.S., or if you can find a civil penalty that might be applicable to their act, you have real means of getting their attention.

  • Change ISPs. My colo company specifically states in our contract that they will not touch my server (physically or remotely) without my prior consent. Once they had to rearrange the rack to fit a new server in, and they called me to ask if I wanted to be present or to move mine myself.

  • Other side (Score:5, Interesting)

    by Spazmania (174582) on Saturday December 26, 2009 @01:40PM (#30556862) Homepage

    On the other side of this, your hosting provider has a guy who keeps angrily reporting mysterious outages where his machine keeps running even though he's on a trivial switch connection like everybody else. The guy then refuses access when they try to figure out what's going on so that they can fix it.

    They shouldn't be rooting your server. That crosses a line. But if I were in their shoes, I'd say: "I'm sorry sir; we've exhausted our diagnostic capabilities without more closely examining your server. Without the root password, there's nothing more we can do for you."

    • They could ask for the logfiles as stated before. Hell, a user account with limited time sudo access would be less invasive, but a copy of relevant logs should do fine.
    • by MBCook (132727)

      When we were in situations like that, before we got our own datacenter and colocated, we would always do the same thing. We'd make sure everything was backed up (for if something got screwed up).

      Then we'd change the root password, and give them the new one. That way they could look around at whatever they needed, change what they needed.

      When they said they were done, we'd change the root password back. They had all the access they needed, but couldn't mess with stuff the rest of the time.

  • by jascat (602034) on Saturday December 26, 2009 @01:43PM (#30556888)
    As someone that works in support for a hosting provider, you're the type of customer that irritates me the most. While they shouldn't be rebooting your box to get root access without your consent, you should at least help them help you. Give them an account with limited sudo access to view your logs. If that won't do, then provide them with the necessary logs. If that's not good enough, don't expect support and move your stuff to some place that doesn't provide the level of support you're paying for.
    • by Sargonas (1111357) on Saturday December 26, 2009 @01:52PM (#30556944)
      Agreed! What you are asking and what you are wanting are an unreasonable combination. Take a step back off your sysadmin high horse ( I am allowed to use that term, since I too was once on one) and look at it from their point of view. You are sending them WTF tickets and at the same time refusing to "help them help you". Honestly, what do you expect?!? Agreed they should not be rebooting your box to get access without first warning you, but at the same time you are demanding a response asap and then withholding critical info from them. What do you expect them to do? As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login. I will bet all the money I have made in my previous career as a sysadmin for several large companies and hosting companies that in your hosting terms it clearly states they own the system, hardware and software, and that you have no inherent right to deny them access. (unless we are talking about a co-located server you personally own, but since you did not state that I can only assume we are not.) In short, you are being a jerk. Get over yourself and either A: work with them to help you, B: diagnose your own damn problems and stop asking them to without giving them the help they need, c: change hosts to someone who more suits your needs, d: colo you own box in an IBX and handle all the work yourself.
      • by http (589131)

        From TFC:

        I will bet all the money I have made in my previous career as a sysadmin for several large companies and hosting companies that in your hosting terms it clearly states they own the system, hardware and software, and that you have no inherent right to deny them access. (unless we are talking about a co-located server you personally own, but since you did not state that I can only assume we are not.

        From TFS:

        With sufficient memory and CPU, I could install VMware and run my entire system within a VM,

      • by hacker (14635) <hacker@gnu-designs.com> on Saturday December 26, 2009 @02:36PM (#30557234)

        "As the above poster said, either create a limited account for them with only log file access, or else man up and just give them a full login."

        I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.

        I find these to be unacceptable terms.

        • by ShinmaWa (449201) on Saturday December 26, 2009 @03:25PM (#30557578)

          You say this

          I can't give them a limited account, because they've locked me out of accessing my own machine, demanding I give them the root password before they hand access back to me.

          ....however, from another post you let the truth slip out

          they moved my drive to a different chassis, with completely different hardware, and are asking for the root password so they can reconfigure everything to coincide with that hardware change (...LATER...) When they migrated it from Savvis to some datacenter in Dallas 2 months ago.....

          So you openly admit the machine IS NOT YOURS. You are essentially keeping them from their own machine, which I find unethical. I can't blame them for taking matters into their own hand and rebooting the system into single-user mode and locking you out until you play nice.

          Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere.

          • by hacker (14635) <hacker@gnu-designs.com> on Saturday December 26, 2009 @03:34PM (#30557660)

            "Stop being a jerk and cooperate with the owners of the machine you are renting or take your data elsewhere."

            Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.

            Basically I am leasing a physical server from company (A) who is leasing it from company (B), and that too may not be the end of the line. (B) may not own it either, and they may be colocating hardware from company (C) or (D) somewhere in there.

            So whose TOS am I subject to here? Who is violating whose laws? It gets curiouser and curiouser the more I dig into it.

            • Re: (Score:3, Insightful)

              by ShinmaWa (449201)

              Apparently it's not their machine either, as they lease the hardware from someone else. I asked them to pull the primary drive in the system and overnight it to me and bill me for it, and they refused, stating that it is leased equipment and they do not own it.

              Okay.. so now you admit you don't even own the DRIVE. Even better. Sorry, but my conclusion is that no matter what agreements your hosting provider may have with others, YOU are the one in the wrong here -- not them.

              Have them burn the data (which you more than likely own) onto a CD/DVD, then host it yourself since you claim to be so much more competent then they are.

  • by casualsax3 (875131) on Saturday December 26, 2009 @01:43PM (#30556894)
    Switch providers. Plenty offer remote reboot and serial console or KVM for both VMs or physical servers, which would allow you to go crazy with custom encrypted partitions etc. At the end of the day though, someone somewhere at the hosting company is still going to be able to reboot your server into a rescue environment and reset the root password. Go colocation if you're really that paranoid about it.

    You also have zero chance with litigation, unless you've somehow gotten them to sign something saying they specifically won't muck around in your server.

    I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?

    • by asdf7890 (1518587)

      At the end of the day though, someone somewhere at the hosting company is still going to be able to reboot your server into a rescue environment and reset the root password. Go colocation if you're really that paranoid about it.

      A good encrypted filesystem setup can be sorted such that nothing can be mounted without your external influence. At this point the host will not be able to get hold of the data from a rescue environment as the keys are external to the server. Of course this means that in a genuine reboot situation (such as a power outage that lasts longer than the UPS can survive) you will have to intervene (providing the keys) to start the services again which will be a hassle if it happens at a bad moment like the middle of the night if you have no support cover who have access to the keys too.

    • by hacker (14635) <hacker@gnu-designs.com> on Saturday December 26, 2009 @03:12PM (#30557506)

      "Switch providers. Plenty offer remote reboot and serial console or KVM for both VMs or physical servers, which would allow you to go crazy with custom encrypted partitions etc."

      They offer KVM access, at $35.00/day, which in this case I refuse to pay to fix what they broke, outside of the context of the server. They migrated me from one chassis to another with completely different hardware, causing my machine to go offline. They want me to pay $35.00 for 24-hours of KVM access to reconfigure the network to support the hardware they moved things to.

      Alternately, they want me to hand over the root password (not a privileged account, but THE root password), so they can do it themselves. Since I installed, configured and manage the OS entirely on this machine, and they've demonstrated their ineptitude before, I'm not giving them root. Ever.

      "I'd also like to know how you *know* it's a hardware or network issue outside of your server. How do you know it's not your NIC driver hanging up? Older e1000 drivers (super common card in the hosting industry) are quite flaky. What research have you done outside of your internal monitoring?"

      Because this server has been running 24x7 for about 3 years without a single outstanding issue. When they migrated it from Savvis to some datacenter in Dallas 2 months ago, I've had no less than 20 separate outages , while the underlying OS and application stack itself has not changed in any way to facilitate those outages.

      In every single case, they demand that I give them the root password, so they can diagnose the issues on the machine. In every single case, I've shown them nagios, ntop, hotsanic, sar, etc. logs demonstrating that the OS itself is not the cause of the outages.

      For example, since this migration to Dallas, every other Sunday between 7:00am and 8:00am EST, my server's load goes over 100 as incoming connections spike over 700/sec., sendmail refuses connections due to the load, and the box seizes up. The logs show that the connections are established and then hang. NOTHING on the machine triggers every other Sunday between these hours that would cause that.

      Only a few days ago, they indicated that the NIC on the server may be causing the issues. I'm down 2-3 hours every other Sunday because of this.

      They're not asking for the logs, they're asking for root. That's a completely separate (and unacceptable) solution to their own problems outside of the box itself.

      • Re: (Score:3, Insightful)

        by don.g (6394)

        Your MTA melting due to incoming connections is not the fault of their network. It's your box. Fix it, or get someone else to, or don't run an MTA (srsly, SENDMAIL? The 90s called, they want their line noise configuration back). If the connections never transfer any data, maybe SYN cookies would help? (is there a full TCP handshake?) Did you get a new IP when you moved?

        And $35 isn't that much to pay. Surely you're paying several times that per month for the hosting, and if not, their margins are thin

  • Dell Drac (Score:2, Interesting)

    by ulzeraj (1009869)
    Password on GRUB will not protect against physical access to a machine. Maybe the best thing you can do is to encrypt the disks. And for now on try to get servers with Drac http://en.wikipedia.org/wiki/Dell_DRAC [wikipedia.org] or something similar installed. Through Drac's remote console you can remotely access the computer during boot process as if you were sitting at the local console.
  • Among the many choices you have, you can install a remote monitoring/administration card.
    But that's really using technology to solve the wrong problem. The problem is your ISP.
    Fire your ISP. You already have two very good reasons for doing so. First, they
    should simply ask for the logs, not demand entry into the system. Second, for taking
    down your server, breaking into it (what if you had data on there you didn't want
    unauthorized people to see?) without your express, positive, verified consent.

    Using technolo

  • "...They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs.."

    What does the word "rooted" mean in this case? They did not have the root password so what is the posted referring to here?

    Disclaimer: I am no computer geek.

  • This is SOP at The Planet - which hosts on the cheapest commodity hardware they can hack together. MiniATX with Celeron procs, all stacked together on a bread rack. The switches are zip-tied to the racks, as are the power strips.

    My NDA has long since expired - I'm open to answering questions via email if anyone has them.
  • Name and Shame (Score:4, Insightful)

    by Charles Dodgeson (248492) <jeffrey@goldmark.org> on Saturday December 26, 2009 @01:53PM (#30556954) Homepage Journal

    If you have some reason that you haven't moved to a different provider, at least let the rest of us know who to avoid. Name and shame, please.

    As others have pointed out

    • If they have physical access, you can make things a bit tougher for them, but never impossible
    • If all they wanted was access to your logs, then create a user for your providers that is in a group that can read your logs
    • Check with your local ISPs to see if you can get a business account (for a static IP address) and self-host. I'm fortunate enough to have FiOS where I live, and while Verizon is really confused about having a business account at a residence, the headache is worth it. I've got about an hour's worth of UPS at home.
    • At least consider the possibility that your diagnosis is wrong. Maybe you've been rooted maliciously and not by your provider. Or maybe what's going on is your own misconfiguration. At least be open to this possibility (and so give them access to your logs to assist in diagnosis).
    • And, of course, consider changed providers.
  • Sue them. And switch to a different company.

  • More details please? (Score:5, Informative)

    by bsDaemon (87307) on Saturday December 26, 2009 @01:53PM (#30556962)
    Are you co-locating a machine you own outright, or do you have a "dedicated hosting" package with the company? I was a system admin at a web hosting company for a long while, and on our dedicated packages if a customer took root access they had to inform us if they changed the root password. We also kept root ssh keys to all of the servers just in case someone wanted to try and be a dick about it. The logic is the machine is actually our property and the customer is renting its use, just as most apartment complexes will keep master keys to the units.

    However, if you own the machine and just have it stuck some place, essentially just paying to rack it and plug into the network, then you may just want to create a limited account that has read permissions on syslog stuff and let them have that for investigative purposes when you need to request access. But, if it's not their machine then they don't need to be shutting you off, booting single-users and rummaging through your stuff.
  • Why did you not just give them access to the logfiles. Just setup a new apache on port 8080, do a few symlinks to bring the log files into the default html folder, update config to follow symlinks, add a .htaccess file and you are done. Should not take more then 20 minutes.

    How exactly did you expect them to help you, if you are the only customer with problems, and you don't give them any access to your log files.

    And it's time to change hosting provider if they really did the "bringing it down and poking aro

  • You can do this (Score:5, Informative)

    by calmofthestorm (1344385) on Saturday December 26, 2009 @01:55PM (#30556982)

    My server does this. The bootscripts for Ubuntu's dropbear package allow you to embed it on the initrd pretty easily, such that this occurs. I had a hard time because our network uses really weird settings (the gateway is outside the netblock and we have nonstandard mtu) and it's surprisingly hard to change this in early boot. Anyway, I'd give this a try; just install the dropbear package (or if not on ubuntu, unpack the deb for it and look at the initramfs scripts, should be easy to adapt to your distro of choice). You can even have a different root password for the initramfs and the real system, or use a keypair.

    If you want a less hackish and more reliable [and expensive] solution look into a remote [power] switch and one of those remote admin cards that basically gives you KVM over network.

  • First: switch providers. Do not put up with this behaviour.

    The only thing you can do otherwise it use encrypted filesystems for your data (you don't need to encrypt *everything* including the root filesystem, just main data store(s) like /home & any databases & sensitive logs stored elsewhere and temporary storage areas) without storing any trace of the keys on the server or anywhere else accessible by the server. Have the server request (or otherwise wait for) the keys to be provided by you before

  • by couchslug (175151) on Saturday December 26, 2009 @02:00PM (#30557006)

    "How do I turn a whore into a housewife?"

    Some things are only solved by replacment.

  • by Animats (122034) on Saturday December 26, 2009 @02:01PM (#30557016) Homepage

    The logs should tell you why the machine crashed.

    How busy was the server?

    There's an ongoing Linux problem with crashing when a program needs more memory, the file cache is using all available memory, and a locking problem prevents paging out a file. Search for "prune_one_dentry" oops (about 4000 hits in Google, from 2002 to 2009). Despite years of patches, this is usually fixed in practice by throwing more RAM at the server. This failure is likely to happen when very large files are open and in use (as with a busy database) and programs are being launched at a high rate (as on an server).

  • Simple:

    1. Require all passwords to be Yubikey OTP passwords on any login prompt.
    2. Refuse access, and only give them the logs manually.
    3. When they shut your server down and open it up to yank the drive, hit 'em with a breach of lawsuit.
    4. ????
    5. PROFIT!

  • How about:

    a) when they ask for root, change it to something innocuous, let them log in and do their stuff, then change it back.

    b) change hosting providers.

    c) host it yourself. Get a business class internet connection to your office/home/etc, rig up a closet with power and A/C. And if you need "five nines", get a second power provider and a UPS, and a second internet connection.

    Ultimately, physical access is everything.

    And yeah, why are you asking for help if you're not going to let them help you? Are you

  • Get a dedicated server running the latest centos or ubuntu server release. Use Xen to run your various applications in dedicated virtual machines. You can encrypt entire domains in a number of ways both internal and external. A dedicated test domain can be set up for your hosting provider to access, etc.

  • Hello, I work for a very similar company that provides support. How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access? That would surprise me greatly because no one in this field would care enough to go through the trouble of a sophisticated hack of your server, and besides, if they could do it, so can anyone else.
    • by hacker (14635) <hacker@gnu-designs.com> on Saturday December 26, 2009 @02:46PM (#30557326)

      "How do they root your box? If your company is like mine, they can't simply reboot the box and log in via singles to gain root access, so how is it possible that they even get in? Are you suggesting that they hack it somehow to gain root access?"

      They have KVM access and forcibly reboot the server, and when it comes back up, they enter it in single-user mode. They've done this at least 3 times before, while I was logged into it, and when the server came back up about 15 minutes later, the lastlog for my own login was missing from the logs. They attempted to clean up the logs to hide their own activities.

  • by Culture20 (968837) on Saturday December 26, 2009 @02:25PM (#30557170)
    Just stop Spamming, and they'll stop rooting you. And don't ask us how to prevent it, because they have physical access. You're hosed. Stop spamming.
  • It sounds like you have a "Managed Server" type of plan with your hosting provider. With a managed plan, a provider has some legal obligations (despite customer instructions) to maintain the host. Go find an "Unmanaged" hosting provider, or colo your own equipment.

  • by Eil (82413) on Saturday December 26, 2009 @02:54PM (#30557388) Homepage Journal

    Okay, since a lot of Slashdotters run their own servers rather than utilize the services of a web hosting company, let me provide some background info. I don't know whether the OP is one of our customers or not, but at the web hosting company I work for, there are two ways to host your server with us:

    1. You can co-locate your hardware with us and purchase a unmanaged plan where the only support we offer is reboots and network troubleshooting. Everything else from the OS to web applications is your sole responsibility.

    2. You can rent a server from us, which comes with full managed support, meaning the box is provisioned and configured by us, and our techs have full root access to your host in order to resolve any problems that come up. All services on the machine are monitored by Nagios, so we know (and react) within 5 minutes when a service stops responding.

    You don't specify which hosting plan you have, but from your description of your problem, it sounds like you purchased #2. All of the things you describe are exactly what our technicians would do if we were charged with keeping a managed server online and a customer was making that task impossible to do. If a customer is asking us to fix a problem and is only making it worse or more difficult by virtue of their incompetence, we have been known to lock them out of their own server until the problem is fixed.

    The bottom line is: don't rent a managed server if you don't want managed service. If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on.

    • Re: (Score:3, Informative)

      by hacker (14635)

      "If you want full control over your hardware, you need to talk to the sales team and tell them that you want an unmanaged plan. The trade-off, of course, is that you have to deal with your own "WTF" problems from then on."

      This IS an unmanaged plan. All the provide is ping and power, I do the rest. I manage the OS, the configuration and everything else. This is not VPS, I lease a physical server, and they don't touch it.

  • by ECXStar (533351) on Saturday December 26, 2009 @03:15PM (#30557520) Homepage
    I host with Softlayer.net (dedicated boxes) and I had the same mysterious issues, server going offline and coming back on. I have a different approach. I trust the techs of the company I'm hosting with so I don't mind giving up root access to chase this problem down. What I do after that is change the root pass again and I'm done. What I'm finding is when the OS and logs come back clean, the problem is mostly likely tied to a DC router issue (a bug or misconfiguration). That's exactly what the excellent techs at SL found. They even filed an RFO (reason for outage) report several days later explaining the problem in detail. So, just like everyone here says, get with a good hosting company and put some trust in the support staff. I used to think that all these companies were about the same level of service if your on a dedicated but, I soon found out you really do get what you pay for.
  • by Anonymous Coward on Saturday December 26, 2009 @03:31PM (#30557628)

    First off, total disclosure - I work for a fairly well know web hosting provider as a system administrator.

    There's basically three plans we have.

    #1 - Managed hosting. We build the box, we manage it, we give you an account to do stuff with. We never give you root. Ever. While I realize the thought of this is anathema to the majority of the slashdot crowd, the bottom line is that webmasters != sysadmin, and there are very few good reasons why a webmaster actually needs root. Obviously in these instances, we can access the machine whenever we want, but as a matter of practice, we don't unless monitoring pops and alert, or a customer submits a ticket. If there's going to be downtime, we try our damndest to work out a time with the customer, but some things (eg, failed drives in an array) constitute bringing the server down without prior customer contact.

    #2 - Unmanaged hosting. We build the box, install whatever OS you want on it, and then turn over root. We do not monitor the box except for ping (and if you firewall off ICMP, we'll turn that off too), and we don't touch the box without a specific request from the customer. If the customer wants us to touch the box, it's a very exorbitant hourly rate (except for hardware failure, as the customer is renting the box from us, we'll replace hardware at no charge, but any work on the server itself outside of that is billable). For these boxes, we would obviously do the same thing with as the OP - we ask for the root password. I'm perfectly ok with providing our public key as well, but most folks would rather just turn over the root password and be done. Occasionally, we do have to root these boxes - either because the customer has forgotten the root password, or because the customer has received a complaint of doing something illegal (like running copyrighted torrents) on the box, and we're forced to investigate to cover our own year. But for the most part, we don't ever want to touch an unmanaged box if we can possibly avoid it. Giving unskilled people root access who break their servers and then want us to fix it is not fun, hence the very large deterrent of the hourly rate. It prevents folks from choosing an unmanaged server just to save a few bucks and then running to us every time something goes wrong.

    #3 - Colocation. You supply the hardware, or you can buy/rent hardware from us. Generally folks will supply their own, and we just drop their network feed into their cage and they take it from there. I can count on one hand the number of times I've had to touch our colo hardware over the years, and if I'm using the right finger, I can make a rude gesture while I'm doing said counting. Generally folks who choose a colo option know what they're doing, and don't need us, and only call if there's an event that's actually beyond their control, like a network issue.

    So honestly, I would take the OP with a grain of salt. If he's got his machine walled off so that only he can touch it on a regular basis, but he keeps opening tickets on a regular basis wanting to know exactly what happened, you're not leaving the hosts tech staff with alot of options. If you're suffering outages, it's a binary question as to who's fault it is - it's either the providers (whether it's network, core internal servers such as DNS, or the like) or it's your servers. Presumably the host is going to know when it's their problem, so if they're asking to take a look at your server, that means the problem is probably actually your server, and not their network. The OP either needs to lose the ego and give up the access or fix his own problems. I suspect that if the OP were to change hosts, the tech staff would not be sorry to see him go

  • by arbiter1 (1204146) on Saturday December 26, 2009 @05:08PM (#30558358)
    Buddy of mine had a box at ovh and he found ssh keys stored in the "/root/.ssh" which can be setup to allow log in without need of the password, he found stored ssh keys in there from them and log's showing someone from the datacenter going in there and poking around. you should check in there to see if there are keys in there and delete them and change all your passwords.
  • Use SELinux (Score:3, Insightful)

    by UnderCoverPenguin (1001627) on Saturday December 26, 2009 @08:45PM (#30559770)

    Enable SELinux in your server. Then disallow root from doing anything but looking at the logs. (Also, create a new, suitably enpowered, account for running your server). Then they can have root access all they want and not be able to mess with your server.

"I got everybody to pay up front...then I blew up their planet." "Now why didn't I think of that?" -- Post Bros. Comics

Working...