Ask Slashdot: How Best To Disconnect Remote Network Access? 284
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
You have got to be... (Score:4, Insightful)
fucking kidding me.
rtfm (Score:4, Insightful)
Short answer? Yes. (Score:5, Insightful)
Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.
But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.
Re:Ever heard of managed switches? (Score:0, Insightful)
Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
Remotely access...not locally...
Got Ethernet? Yes, lots of options. (Score:4, Insightful)
If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.
Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.
Who's woefully inept? (Score:4, Insightful)
> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks
If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.
Re:You have got to be... (Score:4, Insightful)
Oh the irony (Score:5, Insightful)
How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.
Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".
Do yourself a favor and change jobs.
Re:Firewall rules help. (Score:4, Insightful)
Yeah, I don't mean to be rude, but if you have to ask, you probably shouldn't be calling the vendors inept.
Off the mark, missed the target. (Score:5, Insightful)
I think the OP is missing something.
I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.
We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.
The way forward is to merge IT and process control. Unfortunately, that's easier said than done.
check your contracts before doing any thing (Score:5, Insightful)
check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract
Re:Short answer? Yes. (Score:5, Insightful)
A Christmas tree light timer ??? How does the OP have a job?
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.
And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.
-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:
Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
Re:Short answer? Yes. (Score:5, Insightful)
Re:Use a pair of diagonal cutters. (Score:5, Insightful)
And then require the supplier to be on site to do the upgrades to make sure that they do it right. Screw anyone that complains, bring it to the highest level of the organization with hard numbers of how much a stop will cost.
Total isolation of mission critical networks is the only thing that works.
Not your job. (Score:5, Insightful)
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
Re:Short answer? Yes. (Score:2, Insightful)
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
amen to that
Thank You (Score:4, Insightful)
Hats off to you sir (or madam as the case may be.)
The device (Score:5, Insightful)
The device is called a "firewall" and is set up by an "IT Professional"
You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.
Re:Ever heard of managed switches? (Score:5, Insightful)
I am one of those registered engineers who really does understand both the IT and the Operations sides of the issue. Yes, I do process integration for a large utility and yes I live with my creations. Most of you in IT don't have a clue about the operational side of the fence, so please hold your snide comments until you understand the whole issue. Yes, we've seen what remote access follies can do. Allow me to point out that nobody in this business should be pushing patches to the plant floor. Remote firmware updates are reckless activities that deserve to be prosecuted for malpractice.
My employer has seen a few idiot project managers who, despite warnings from staff, contracted companies who demanded remote access. Suffice it to say that these people will not make such mistakes again.
In an office, there is usually a warm body at the other end of the keyboard. They can be instructed to do things. In any case, the product is data which can be backed up and restored if needed. If you chose to push patches in a situation like that, you could trust the end users to call you if something goes sideways. However firmware in a substation or in a controller is really not meant to be updated remotely. You should be standing there just in case you need to run things manually or need to shut down certain devices first. These places do not normally have people present to call if something doesn't work.
So when a vendor demands remote access to your substation or large asset, the answer should be an emphatic NO! and WTF? and "I'm taking my business elsewhere."
There is no good way to push a patch in to a control system. Those of you who think pushing patches is good need to come with me and clean up the messes that result from such behavior. You need to realize that software and data is not the end product here. There are no backups. There is only real product, real energy, and real messes when something fails. And if someone is hurt or killed, well, limbs and lives can not be backed up and replaced. If you're still throwing patches at the wall in the hope that nothing goes wrong, you are not welcome on the plant floor.
Re:Ever heard of managed switches? (Score:4, Insightful)
And that's what I find odd about the OP's request. Why is it an issue requiring a technical equivalent of hiding the car keys from the children? Surely the person in the company who allowed RA on the production line is sacked, and the supplier who pushed the updates has been replaced? Those were the actual problems. What else needs to be done? Maybe explain to the replacements why they are the replacements.