Ask Slashdot: How Best To Disconnect Remote Network Access? 284
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
Get another job? (Score:3)
Re:Get another job? (Score:5, Informative)
crontab:
Ever heard of managed switches? (Score:5, Interesting)
Re:Ever heard of managed switches? (Score:5, Insightful)
I am one of those registered engineers who really does understand both the IT and the Operations sides of the issue. Yes, I do process integration for a large utility and yes I live with my creations. Most of you in IT don't have a clue about the operational side of the fence, so please hold your snide comments until you understand the whole issue. Yes, we've seen what remote access follies can do. Allow me to point out that nobody in this business should be pushing patches to the plant floor. Remote firmware updates are reckless activities that deserve to be prosecuted for malpractice.
My employer has seen a few idiot project managers who, despite warnings from staff, contracted companies who demanded remote access. Suffice it to say that these people will not make such mistakes again.
In an office, there is usually a warm body at the other end of the keyboard. They can be instructed to do things. In any case, the product is data which can be backed up and restored if needed. If you chose to push patches in a situation like that, you could trust the end users to call you if something goes sideways. However firmware in a substation or in a controller is really not meant to be updated remotely. You should be standing there just in case you need to run things manually or need to shut down certain devices first. These places do not normally have people present to call if something doesn't work.
So when a vendor demands remote access to your substation or large asset, the answer should be an emphatic NO! and WTF? and "I'm taking my business elsewhere."
There is no good way to push a patch in to a control system. Those of you who think pushing patches is good need to come with me and clean up the messes that result from such behavior. You need to realize that software and data is not the end product here. There are no backups. There is only real product, real energy, and real messes when something fails. And if someone is hurt or killed, well, limbs and lives can not be backed up and replaced. If you're still throwing patches at the wall in the hope that nothing goes wrong, you are not welcome on the plant floor.
Re:Ever heard of managed switches? (Score:4, Insightful)
And that's what I find odd about the OP's request. Why is it an issue requiring a technical equivalent of hiding the car keys from the children? Surely the person in the company who allowed RA on the production line is sacked, and the supplier who pushed the updates has been replaced? Those were the actual problems. What else needs to be done? Maybe explain to the replacements why they are the replacements.
Re: (Score:2, Troll)
Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
Remotely access...not locally...
what the fuck man? get another job which is the first post is the only thing that pops to mind. IF you cannot be bothered with a physical button and want to be able to turn the ports on/off remotely then get a device which allows you to do that - commonly known as a managed switch.
or get a fucking pc with two network cards. it's all the same anyhow and can be used remotely to turn an interface up or down or scripted.
Re: (Score:3)
> Connect the VPN device to the locked down port.
Finally someone with a solution. The job of manufacturing is to make things, and more often than not that will require vendor support. Security is nice, but performance is more important. Judging by the OP's words, going through IT is major pain, so they don't.
I would recommend a VPN or a web based solution to enable access with a simple password. It has to be easy to use for the vendor - they have many customers, and remember that they have corporate mach
Firewall rules help. (Score:5, Informative)
iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.
Re:Firewall rules help. (Score:4, Insightful)
Yeah, I don't mean to be rude, but if you have to ask, you probably shouldn't be calling the vendors inept.
You have got to be... (Score:4, Insightful)
fucking kidding me.
Re:You have got to be... (Score:5, Informative)
Re:You have got to be... (Score:4, Insightful)
picking the right tool for the job (Score:3)
Programming a solution to a solved problem is overkill.
In this case I believe it's very appropriate. They have a static arrangement, (vendor wants in, someone turns access on, manually) and when they're done, someone's supposed to shut it off. This process has demonstrated a history of being unreliable. So the solution comes down to one of three things. (1) replace or retrain whoever is in charge of the process in the hopes of improving reliability, (2) automate the process that is not being done relia
Re: (Score:2)
My thought exactly. This is such a basic network admin function...
Re: (Score:2)
we usually bang in the nails with books. they do get broken quickly, so we are looking into more robust solution. we could take wooden planks, but that does not seem like a very good solution. maybe we can use rocks or something ?
ps. this article must be a joke to see what silliness can get through on slashdot :)
Re: (Score:2)
You must be new here.
Now to say something more helpful: good luck getting the vendor to agree to anything. The equipment has been sold, signed, and delivered. Whatever contract was put in place by the CEO over golf, that's what you get to try and work with.
But is it worth turning into a BOFH just to screw the manufacturing guys?
If the problem is really as bad as it sounds, maybe it's time to start looking for another job.
rtfm (Score:4, Insightful)
Quality of questions very low.. (Score:3)
Short answer? Yes. (Score:5, Insightful)
Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.
But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.
Re: (Score:2)
You do not need to think of any of that if you just stick a managed switch between the internet connection and the equipment.
Enable the port when you want them to have access and disable the port the rest of the time.
Why is this complicated? Why is it a question even?
A Christmas tree light timer ??? How does the OP have a job?
Re: (Score:2)
How does the OP have a job?
The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.
Re: (Score:2)
Agreed.
Put in a managed switch, log in the switch to enable / disable the port when you want... <yawn> ...
Re:Short answer? Yes. (Score:5, Insightful)
A Christmas tree light timer ??? How does the OP have a job?
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.
And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.
-_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:
Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
Re: (Score:2, Insightful)
If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.
amen to that
Cost center (Score:3)
You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...
Umm, IT actually IS a cost center in most companies. My company is a manufacturing firm and I run all our IT operations (among many other duties) and our customers do not pay us a dime for our IT. They pay us to deliver the products we make for them. IT falls in to the necessary/useful expense category. If you are not being paid for the IT then it is by definition a cost center. Lot's of necessary, useful and important things are cost centers - it's not a pejorative. The job of IT is to help the other
Re: (Score:3)
Well, when I say "cost center", I mean they look at IT resources the same way they look at paper towels for the bathroom... a necessary expense you should spend as little as possible on. Unfortunately, that's what they teach MBAs and the like in college. Doubly-unfortunate, they listen and never bother thinking on their own, or listening to people in the field.
The difference between the cheapest solution and the second cheapest solution is usually the difference between "will work 4 out of 5 times" and "wil
Re: (Score:2)
Oh, and quit reading slashdot cold turkey. That's part of your problem.
Re: (Score:3)
I have never seen anyone fired for doing his job well. I have seen people fired for being insubordinate and abusive (though not often enough!). Are you sure you know the difference?
Yes. I have 14 years of industry experience. I've seen more than you, kid.
Re:Short answer? Yes. (Score:5, Insightful)
Re: (Score:2)
Agree. They will do this. Seen it everywhere. And if the run is too long for a cable, prepare for wireless.
~Sticky
Re: (Score:2)
That, or you might find an el cheapo four port switch between a production machine and its normal port where there wasn't one before, so unless you set the max MAC addies to one as a matter of daily business, you may have an ugly surprise.
Re: (Score:2)
Have the connection closed with a metal case and a key, when the time comes the company doing the
DIY (Score:3)
Re:DIY (Score:4, Funny)
You could also do it in Visual Basic, with the added advantage that you could create a GUI to trace their IP address.
DCS network should be totally isolated (Score:5, Interesting)
I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.
Thank You (Score:4, Insightful)
Hats off to you sir (or madam as the case may be.)
A futile effort (Score:2)
let them fail. its the beauty of capitalism (Score:2, Flamebait)
your are describing managers who are incompetent and not qualified to run their business.
you have done what you can do. you arent the company president.
this is like dealing with a durg addict. you cant save them. you cant change them. they have to want to change, and they dont want to.
let them fail. let them go bankrupt.
make sure you cover your ass. keep documentation of their stupidity and your warnings.
and keep your resume updated.
Got Ethernet? Yes, lots of options. (Score:4, Insightful)
If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.
Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.
Routers can do it (Score:2)
I don't know what type of router you have but many do have scheduling capabilities. Actually publishing information like brand and model of router would be pretty dumb.
My first step would be to contact your router manufacturer and if necessary get one that has that capability. You could even put all of your manufacturing equipment behind one unit on it's own segment of the network with limited access from outside, assuming that you really need network access at all.
Unplugging from the network is an option
Re: (Score:2)
I don't know what type of router you have but many do have scheduling capabilities.
BINGO.
A freaking consumer-grade Netgear router does this!!
(I can verify first-hand, as I just bought one about a week and a half ago.)
Lots of solutions ... (Score:2)
You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything i
Who's woefully inept? (Score:4, Insightful)
> As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks
If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.
Oh the irony (Score:5, Insightful)
How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.
Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".
Do yourself a favor and change jobs.
Passwords (Score:2)
Leave it connected to a separate secure network. (Score:2)
Either use managed switches to separate the LAN's and lock down the ports to only allow certain mac addresses on the PCS VLAN, or create another LAN with dumb switches. Add a VPN box that is connected to both networks, make it the only allowed method of connecting remotely to the PCS LAN, give the vendor a VPN account, and only enable the account temporarily when it has been approved.
Off the mark, missed the target. (Score:5, Insightful)
I think the OP is missing something.
I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.
We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.
The way forward is to merge IT and process control. Unfortunately, that's easier said than done.
Re: (Score:2)
Most know, some fight to get security put in place...
But management in-house and managers at the customer tend to view security as a needless expense. Mostly because they have 'a firewall' (non specific...) and believe that one layer of security is plenty. Especially since 'the vendor promised it was 100% secure'.... sigh
Oil rig PCS network 'secure plant network' that goes onshore office network internet.
Since the firewalls 'are secure' the management think there is no even theoretical way for anyone to
Re: (Score:2)
The IT side and process control side should each pick someone to go and learn how the other side works for a while. A quick tutorial on basic networking concepts, a second on network security and infrastructure set up, and then actually being the PFY or Intern for a few days. The IT designate would go to process control and do similar.
Basically, each side has to have someone who at least has an idea as to what goes on on the other side.
Re: (Score:2)
check your contracts before doing any thing (Score:5, Insightful)
check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract
Time-restricted firewall rules (Score:2)
You are looking for time-restricted firewall rules.
You can roll them yourself (Linux, Free-BSD), by just having two sets offirewall rules, and switching to the restricted set after the time expires. If you re-inilialize or don't use connection tracking, existing connections get cut. Reloading a firewall rule-set does not cut connections if it does not take too long). You can also isolate this in a specific sub-chain, and then just reload that one to enable or disable the specific connection. That way you ha
health Care systems are some times like this (Score:2)
health Care systems are some times like this in where out side suppliers and vendors have control over systems and some times they don't install updates / say you can't over firewall this system.
Sounds like what is needed... (Score:5, Interesting)
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.
As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)
As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.
Re: (Score:2)
...is a post incident review with support people involved, and their management teams, along with directors and executive involvement
There are some useful training materials from Homeland Security on this. See the National Infrastructure Protection Plan [dhs.gov] One of the key points there is to focus not only on prevention, but fast recovery. You may want to have spare control units on site which can be swapped in if the main ones are corrupted, for example.
Re:Sounds like what is needed... (Score:4, Interesting)
At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication
I work for the "vendor" side of the equation. If we make any changes to a customer system outside this explicit, per-case authorization, we lose any limitations on liability. If we caused a downtime like in TFA, we'd be liable for up to $infinity in lost revenue, overtime, and other damages.
As Rusty says, OP absolutely, positively needs to have a change control process with teeth if it's not followed. If his organization's support contract lets the vendor off the hook for this, they got taken for an expensive ride.
Sounds like a teachable moment (Score:2)
Web Controlled Power Switch (Score:2)
http://www.digital-loggers.com/lpc.html [digital-loggers.com]
Eight power jacks that can be independently controlled over your network. You can control access to the entire device or individual sockets with multiple users and passwords, and they have built-in scripting functionality that shut off sockets based on the time, power-cycle if a repeating ping test ever fails to get a response, and other options I haven't bothered to look into. A real party. I think they're about $
Get a lawyer, not a switch (Score:5, Interesting)
The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.
This isn't a technology problem.
Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.
Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.
Relays & ATtiny (Score:2)
http://www.sainsmart.com/8-channel-dc-5v-relay-module-for-arduino-pic-arm-dsp-avr-msp430-ttl-logic.html?___store=en&___store=en [sainsmart.com]
Then, connect it to an ATtiny - ATtiny85 should be fine; it's only a couple of bucks - and program it to go on and off as you wish. Side benefit: it runs on batteries in case the power goes off.
If you submit the idea to Make, I get credit.
Re: (Score:2)
No. What happens to equipment or people if lightning strikes nearby, or if a major pump shorts out? Will it transmit the current into the process switches, causing a larger issue? Will it electrocute someone nearby? Questions like these need to be answered before tossing equipment into an industrial environment.
Neat idea, needs more than just an ATtiny. It was good though that you picked a relay that requires power from the ATtiny to turn on, I've seen other guys accidentally set stuff to fail open when t
Never heard of a firewall? (Score:3)
All Internet connections must cross a Firewall. Disable inbound connections, done.
Seriously, this is a question?
Re: (Score:3)
Re: (Score:2)
You have never heard of change management?
When access is needed, and authorized/approved, the firewall team opens it.
When the work is done, the firewall team removes the permission.
Is that really a difficult concept?
Re: (Score:2)
In IT, it's a very easy concept. Process control and industrial control systems is another matter entirely. They don't have a firewall team, or an IT staff, or a network admin, or a Windows Domain Architect, or any of that stuff. They don't have 4 days to wait for a change control board to approve access, because they usually need the vendor to fix crap immediately, or lose a few hundred thousand dollars in lost product.
They have Steve, who has been at the plant since God stopped by for tacos. Steve knows
Re: (Score:2)
If they understand the concept of guards at the gate checking people and calling in to make sure visitors were expected, they understand the concept of firewalls.
It doesn't matter if these guys do not have any idea what good IT practices are. If it's multimillion dollars, this is a corporate function. All they have to do is call this number, and say "open sesame". And when done, say "close sesame".
The corporate IT firewall guys can do the magic.
If you'd read my report... (Score:2)
You wouldn't have an osm to worry about.
~Sticky
VPN and Change Control (Score:2)
Not your job. (Score:5, Insightful)
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
Re: (Score:2)
This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.
Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.
By all means give people support in doing their jobs, but don't do their jobs for them.
I agree. IT ISN'T YOUR JOB.
It sounds like your Process control department has outsourced support to the system to this particular company. They are likely monitoring it and this requires always-on access. When a problem does occur, it's up to management to enforce any vendor SLAs that were negotiated and any relevant penalties.
I'm a network engineer for a large company and we also have a separate manufacturing group that has it's own IT. The best that you can do is make sure that there is a firewall bet
Re: (Score:3)
Yup. I'm a controls engineer, and 1. I don't expect IT to either fully understand the nuances distributed controls architecture, or 2. do my job for me. I don't fix accounting network gear, they don't fix my DCS. Knowledge of how to manage TCP/IP networks on the factory floor is Engineering's job, not IT's.
That said, it sounds like OP's engineering manager is a fucking idiot that needs to be canned, along with the staff that tolerated it. This is how refinery explosions happen. This sort of laziness and lac
Wrong approach. (Score:3)
The I told you so and not our problem method is not what's going to get the right result here. What you've done is passed the blame but not fixed the underlying cause of the problem which is a fundamentally flawed approach to dealing with your systems.
Both the process control team and IT have experience that the other team doesn't have. What is needed is for both teams to sit down and nut out what a site wide acceptable policy for ALL gear is and then have the upper level of management sign off on the polic
This is a common issue (Score:3, Informative)
Ignore the haters, they don't understand the politics for this. I used to design industrial Ethernet networks for a large vendor, and we spent quite a bit of time pointing out to customers how dangerous the direct lines were. However, IT departments have very little say over manufacturing networks. This isn't always a bad thing (see the many IT/help desk horror stories). Because the remote access is often required as part of the maintenance contract, offer to partner with manufacturing to install a small firewall with access filters that are controlled by IT, but set (requested) by manufacturing.
A small Cisco ASA, Juniper SRX or its like will do the job nicely, and can shield you from hack attempts along that access path.
Not a technical problem (Score:2)
This is a problem with the organization of your business and cannot be fixed with a technical solution.
"the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this"
You need to raise the risks to management and if they choose to override your recommendation then the problem is theirs, not yours.
Not a Technology Problem (Score:2)
You said they need remote access and botched the upgrade. There is no technological solution for this. If your management insists an incompetent vendor have such access and thinks you can do something about this result then the solution is to find another job. If you aren't being told to fix it, realize that it's not your problem.
You can't fix stupid; you either have to accept it or escape it.
Re: (Score:2)
But management is usually gauged, by their superiors or the board of directors, on results. A major production outage is not a good result. This can be used as leverage to get the process changed, if one has the guts to pursue it.
not just manufacturing (Score:3)
> but reality is that IT gets overridden by the Process Control department in a manufacturing business
It happens in a lot of industries. We're forever chasing vendors who think it's ok to pull our systems out from under us to apply updates, sometimes (thankfully rarely) bricking the systems keeping us down until they can make physical repairs.
I don't think there's a surefire technical solution. We disallow access from outside directly to our hardware via our firewall (the best solution -- don't think christmas tree timer, think firewall or switch controls) but since the outsourcing, our firewall is itself under management from an outside group (albeit a different one) and they don't seem to know what they're doing, except to call an operator to press the reset button when a problem is reported.
But the point is, the problem is a social one, not a technical one. I know you haven't had good results so far, but this needs to be fought in management, not in technology. A major production outage gives you fuel -- get riled up, and go talk to some people. Make it plain that the next time the vendor makes any change at all without first approval from a cross-department board, will be the last act that particular vendor does in your company. Put some teeth in your service contract. Hop to it. Your company is at stake.
The device (Score:5, Insightful)
The device is called a "firewall" and is set up by an "IT Professional"
You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.
Inept much? (Score:2)
pwnnndd (Score:2)
-keyboar battt dead
Top 10 (Score:2)
Easy. (Score:2)
"Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network?"
Yep, it's called a router.
Contract? (Score:2)
Seems to be something that needs to, at least in part, be governed by a contract with the supplier. You might find yourself the scapegoat if off your own back you buile q Wallace & Gromit inspired steam-powered Kill'O'Network contraption.
ELEGANT! ELEGANT! (Score:2)
ELEGANT! ELEGANT!
Sheesh. IPTables, Cisco timeout functions...no imagination whatsoever.
Write an app for a dedicated iPhone (hooked to a power adapter) that is interfaced with an RJ45 switch (low-voltage control circuit for a small relay in place of the usual rotary switch) for the ethernet cable , so that the vendor has to call that iPhone and maintain a connection to keep the ethernet connection active. Limit the phone number sharing to vendors only.
IT jobs... (Score:2)
No wonder some people have a hard time finding jobs, since a lot seem to be taken by cheap labor with unrelated or irrelevant knowledge or workforce repurposed from other departments to cut the costs of hiring someone who has the proper knowledge. I am not in IT, never was, never will, yet even I know of the device the poster seeks with multiple jacks and connection handling functions, which are magic boxes brought by blue
firewall (Score:2)
Why do you want to disconnect after a set time interval? Your time will never be correct. Sometimes, legit maintainance might take longer (e.g. if they're monitoring the system to trace a problem) and often it will be much shorter.
Without knowing the details, what you need is a point inbetween that IT Security controls, and the procedure that says enabling remote access requires form 123a filled out. Which would be a simple paper saying "please enable remote access for vendor X on (datetime) until (datetime
Why is this so hard? (Score:3)
If somebody wants remote access to manage "their" system, these are a few of the things you should insist on:
a) Explicit Contract statements describing the methods of access permitted, when it's permitted and by whom.
b) Contract must spell out the type of testing or diagnostic work to be performed by vendor technicians and who on your company side will pair up with them to validate. The buddy system is to be used at all times when the vendor is gerfinkerpoken mit das machinen.
c) Managed Access, only allow them in on incidents of failure for diagnostic work or for upgrades again, pairing somebody inside your org up with them during the access window. No Carte Blanche access at all, no VPN tokens and no dial up. All access must be over a minimum of TLS 1.1 links, TLS 1.2 is preferred.
d) Penalties in the contract for fucking up. Make it so nasty that if the fuck up they'll pay through the nose if they take you down. Sorry it has to be this way.
e) Specify that you require test results and approval of the test results (including how they tested) before any upgrades. Also provide a test infrastructure or subsystem to allow the vendor to deploy to first to verify that what they're saying actually works. Just because they've done testing doesn't necessarily mean it will work with your hardware and your environment.
f) All workers from the vendor doing the work must be based in your country and be primary to the vendor. No third parties are to have access to your infrastructure or systems. Don't let subcontractors who don't know your systems and processes in.
g) During upgrades or problem sessions a hot call is established to let key stakeholders in your company know what's going on and to provide progress updates on rediation or the success/failure of the situation. Keep your management in the loop and make sure they're aware of the scope of any changes.
h) Backout plans must be provided in case of failures of any changes. I realize this may not be possible with some PLC/Process systems but that should also be a consideration when purchasing these kinds of systems.
i) Maintain air gaps at all times between PLC equipment and any network infrastructure that has access to the Internet or Corporate Intranet. No connections to networks with "office" applications or information flow should be allowed anywhere near your process control networks. The exception to this is when the vendor is troubleshooting or upgrading systems or obtaining log data in accordance with the rest of this.
Re: (Score:2)
What Tastecicles says.
Airgap is the only sure way. I know your system isn't classified, but TREAT IT AS IF IT IS.
Inform your boss -- IN WRITING -- that there is to be no internet connectivity for your PCS. If he won't (or higher power's won't let him) agree, then you have to be prepared to either walk, or face the consequences when someone fucks up your system from outside.
Re: (Score:2)
Follow-up. Again, treating your system as if it was classified...
If you need to apply updates, then get them from the internet, and burn to CD-R and finalize (*not* CD-RW).
Re: (Score:2)
Airgaps aren't a panacea. USB keys, CDs, even floppy disks (yes, these places still have those) can all bridge an airgap in a non-detectable manner.
Most of these systems have no actual monitoring to ensure that the integrity of the network stays constant. And, if it makes a process control professional's life easier, they WILL connect it to the internet for 'a little while', go home, forget, and completely deny they did it if the fit hits the shan.
The people need change too.
~Sticky
Re: (Score:3)
The Iranians had airgaps for their centrifuges...
Security is a layered process. Airgaps do help a lot, but then you have to beef up your physical security.
I'll give one example. There was one company that I did an unpaid internship during my college days whose guys gave me a tour. They bragged about their mantraps, their electronic access control mechanisms, and what measures they had in place. I pointed out that the manual override lock on the door was one that was fairly easy to bump, so unless someon
Re: (Score:2)
> At the least, a civil suit for any damage caused.
Good point. I assume there is a contract, and the contract specifies damages caused by incompetence? In that case it would not be the OP's problem again. The question is whether it is worth suing a supplier - usually that does not exactly help the business relationship. But it may be leverage to come to some kind of resolution.
Re: (Score:2)
actually, I didn't need to, I've dealt with several cases involving computer crime [legislation.gov.uk]. Such activity as the OP describes *may* fall under Section 3.
Re: (Score:2)
I suspect parent is not in the US, so it's not the CFAA. In the UK, it *is* what he called it.
Re: (Score:2)
That's not an "anarchist quote", it's a quote from 1984, a book about totalitarianism. How dumb do you have to be to think that the only alternative to totalitarianism is lawlessness and anarchy? And even if the guy were an anarchist, anarchy rejects hierarchical, centralized state power, not voluntary associations and self-governance.
Re: (Score:2)
Adding a wireless device to such an environment seams a really bad idea!
It's not that difficult to disable wifi and go with wired-only.
Re:Poor mans solution... (Score:5, Interesting)
Our salescritter had jokingly told this same customer, "This system can do everything except make your coffee in the morning." The customer took that as a challenge, and the next time we were there we found that he had set the system up so that when he badged in the front door for the first time in the morning it would fire a relay that would turn the coffee pot in his office on.
Re:Use a pair of diagonal cutters. (Score:5, Insightful)
And then require the supplier to be on site to do the upgrades to make sure that they do it right. Screw anyone that complains, bring it to the highest level of the organization with hard numbers of how much a stop will cost.
Total isolation of mission critical networks is the only thing that works.
Re: Use a pair of diagonal cutters. (Score:5, Interesting)
In addition to requiring then to be onsite, negotiate cost of the onsite support in advance. Include a bonus if everything goes according to plan, and if it doesn't, the vendor is covering the extra cost to put it right.
Re:Use a pair of diagonal cutters. (Score:4, Funny)
Parent did mention putting the USB stick in a person. That sounds both painful and inconvenient.