Ask Slashdot: How Best To Disconnect Remote Network Access? 284
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
Ever heard of managed switches? (Score:5, Interesting)
DCS network should be totally isolated (Score:5, Interesting)
I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.
Sounds like what is needed... (Score:5, Interesting)
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.
As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)
As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.
Get a lawyer, not a switch (Score:5, Interesting)
The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.
This isn't a technology problem.
Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.
Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.
Re:Poor mans solution... (Score:5, Interesting)
Our salescritter had jokingly told this same customer, "This system can do everything except make your coffee in the morning." The customer took that as a challenge, and the next time we were there we found that he had set the system up so that when he badged in the front door for the first time in the morning it would fire a relay that would turn the coffee pot in his office on.
Re:Sounds like what is needed... (Score:4, Interesting)
At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication
I work for the "vendor" side of the equation. If we make any changes to a customer system outside this explicit, per-case authorization, we lose any limitations on liability. If we caused a downtime like in TFA, we'd be liable for up to $infinity in lost revenue, overtime, and other damages.
As Rusty says, OP absolutely, positively needs to have a change control process with teeth if it's not followed. If his organization's support contract lets the vendor off the hook for this, they got taken for an expensive ride.
Re: Use a pair of diagonal cutters. (Score:5, Interesting)
In addition to requiring then to be onsite, negotiate cost of the onsite support in advance. Include a bonus if everything goes according to plan, and if it doesn't, the vendor is covering the extra cost to put it right.