Forgot your password?
typodupeerror
Networking IT

Ask Slashdot: How Best To Disconnect Remote Network Access? 284

Posted by timothy
from the no-soup-for-you dept.
An anonymous reader writes "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network? The why? We are dealing with a production outage of large industrial equipment. The cause? The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days. Obviously the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this and told it was a horrible idea to allow remote access all the time. They were warned many times to leave the equipment disconnected from remote access except when they were actively working with the supplier. Either they forgot to disconnect it or they ignored our warnings. The question is, is there a device that will physically disconnect a network connection after a set time? Yes, we could use a Christmas tree light timer hooked up to a switch or something like that but I want something more elegant. Something with two network jacks on it that disconnects the port after a set time, or even something IT would have to login to and enable the connection and set a disconnect timer would be better than nothing. As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks and repeatedly make blunders like connecting process control systems directly to the internet, use stock passwords for everything, don't install antivirus on windows based control computers, etc. How do others deal with controlling remote access to industrial systems?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: How Best To Disconnect Remote Network Access?

Comments Filter:
  • by gagol (583737) on Friday June 07, 2013 @10:14PM (#43943479)
    Cant think of anything else...
    • Re:Get another job? (Score:5, Informative)

      by msauve (701917) on Saturday June 08, 2013 @06:35AM (#43945209)
      Or use this job

      crontab:

      #turn off at 5 PM everyday
      00 17 * * * /usr/bin/snmpset -v 2c -c private ethernetswitch.example.com IF-MIB::ifAdminStatus.<portnum> i down

      #turn on at 9AM weekdays
      00 9 * * 1-5 /usr/bin/snmpset -v 2c -c private ethernetswitch.example.com IF-MIB::ifAdminStatus.<portnum> i up

  • by Fallen Kell (165468) on Friday June 07, 2013 @10:16PM (#43943489)
    Enable port security which ties each port to a mac address of the other device connected to it so that all ports on the network switch are locked down to just the devices white-listed to connect. Write down what port your gear is connected to which you want to limit access to the internet, and then simply disable or enable that port to allow it to connect.
    • by Anonymous Coward on Saturday June 08, 2013 @09:06AM (#43945851)

      I am one of those registered engineers who really does understand both the IT and the Operations sides of the issue. Yes, I do process integration for a large utility and yes I live with my creations. Most of you in IT don't have a clue about the operational side of the fence, so please hold your snide comments until you understand the whole issue. Yes, we've seen what remote access follies can do. Allow me to point out that nobody in this business should be pushing patches to the plant floor. Remote firmware updates are reckless activities that deserve to be prosecuted for malpractice.

      My employer has seen a few idiot project managers who, despite warnings from staff, contracted companies who demanded remote access. Suffice it to say that these people will not make such mistakes again.

      In an office, there is usually a warm body at the other end of the keyboard. They can be instructed to do things. In any case, the product is data which can be backed up and restored if needed. If you chose to push patches in a situation like that, you could trust the end users to call you if something goes sideways. However firmware in a substation or in a controller is really not meant to be updated remotely. You should be standing there just in case you need to run things manually or need to shut down certain devices first. These places do not normally have people present to call if something doesn't work.

      So when a vendor demands remote access to your substation or large asset, the answer should be an emphatic NO! and WTF? and "I'm taking my business elsewhere."

      There is no good way to push a patch in to a control system. Those of you who think pushing patches is good need to come with me and clean up the messes that result from such behavior. You need to realize that software and data is not the end product here. There are no backups. There is only real product, real energy, and real messes when something fails. And if someone is hurt or killed, well, limbs and lives can not be backed up and replaced. If you're still throwing patches at the wall in the hope that nothing goes wrong, you are not welcome on the plant floor.

      • by FatLittleMonkey (1341387) on Saturday June 08, 2013 @11:20AM (#43946563)

        Suffice it to say that these people will not make such mistakes again. [...] So when a vendor demands remote access to your substation or large asset, the answer should be [...] "I'm taking my business elsewhere."

        And that's what I find odd about the OP's request. Why is it an issue requiring a technical equivalent of hiding the car keys from the children? Surely the person in the company who allowed RA on the production line is sacked, and the supplier who pushed the updates has been replaced? Those were the actual problems. What else needs to be done? Maybe explain to the replacements why they are the replacements.

  • Firewall rules help. (Score:5, Informative)

    by Anonymous Coward on Friday June 07, 2013 @10:17PM (#43943495)

    iptables lets you specify times if you're using a linux box as the firewall, otherwise consult the fine manual that came with your equipment or consult a professional with said equipment. This is bog standard.

  • by vuke69 (450194) on Friday June 07, 2013 @10:18PM (#43943507)

    fucking kidding me.

    • by Pubstar (2525396) on Friday June 07, 2013 @10:27PM (#43943575)
      This. I just got my CCNA and I knew ages ago that there is a time out option in the GUI settings for most Cisco gear . I can't remember the CLI commands, but if I can do it with almost no field experience, the OP should be able to too.
      • by The Mighty Buzzard (878441) on Friday June 07, 2013 @10:37PM (#43943647)
        Barring that it's a one or less cup of coffee bash script write on a linux firewall box. Either write it as a very minimal daemon or run it as a cron job.
    • by asmkm22 (1902712)

      My thought exactly. This is such a basic network admin function...

    • by richlv (778496)

      we usually bang in the nails with books. they do get broken quickly, so we are looking into more robust solution. we could take wooden planks, but that does not seem like a very good solution. maybe we can use rocks or something ?

      ps. this article must be a joke to see what silliness can get through on slashdot :)

      • by dch24 (904899)
        Re; "article must be a joke" ...

        You must be new here.

        Now to say something more helpful: good luck getting the vendor to agree to anything. The equipment has been sold, signed, and delivered. Whatever contract was put in place by the CEO over golf, that's what you get to try and work with.

        But is it worth turning into a BOFH just to screw the manufacturing guys?

        If the problem is really as bad as it sounds, maybe it's time to start looking for another job.
  • rtfm (Score:4, Insightful)

    by jfalcon (163956) on Friday June 07, 2013 @10:18PM (#43943511) Homepage Journal
    There are some firewall/access devices/content filters that restrict access on both time schedules and destination. Maybe talk to your network administrator?
  • by whoever57 (658626) on Friday June 07, 2013 @10:19PM (#43943513) Journal
    Obviously, just put the device behind a firewall. If the firewall operates in bridge mode, it won't use NAT, so the people who insist that their equipment is directly connected to the Internet won't know that it isn't.
  • Short answer? Yes. (Score:5, Insightful)

    by Shoten (260439) on Friday June 07, 2013 @10:20PM (#43943517)

    Part of this depends on how they have remote access...is it dial-in? Are they connecting to a jump host via IP connectivity? Is it a VPN? The solution depends on which of those they use, because it's all different. You can use a relay to open/close the actual circuit to the phone line if they dial in; I know a few power companies that use this as a safeguard for their power substations that have dial-up access. If it's a jump host or VPN, then the details of that solution define the approach.

    But here's a question for you...what about having a limited time to have remote access would have kept this from happening? From what it sounds like, the process control people would have let them in anyways. And then...what happens if they run out of time, halfway through whatever they're doing? Or even more interestingly, what if they screw everything up (again) but then blame it on being disconnected while they were in the midst of doing something, so they can put the blame on you? This sounds more like a people problem than a technology problem.

    • by Dishevel (1105119)

      You do not need to think of any of that if you just stick a managed switch between the internet connection and the equipment.
      Enable the port when you want them to have access and disable the port the rest of the time.
      Why is this complicated? Why is it a question even?
      A Christmas tree light timer ??? How does the OP have a job?

      • How does the OP have a job?

        The real IT guy was fired and replaced with a mid level sales manager. Anonymous submitter for a reason. And maybe they sell Christmas tree light timers and happened to have a few lying around.

      • by ryanw (131814)

        Agreed.

        Put in a managed switch, log in the switch to enable / disable the port when you want... <yawn> ...

      • by girlintraining (1395911) on Friday June 07, 2013 @11:55PM (#43944007)

        A Christmas tree light timer ??? How does the OP have a job?

        You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...

        I just got done with a contract at a large bank (It's one of the 50 largest companies in the United States)... all their deployments are run off USB drives hung off servers at their retail locations, they have 512kbit backhauls to their corporate locations, run DHCP over the WAN, have no QoS, and I kid you not -- about 5% of the managed switches have been forced to 10mbit half-duplex.

        And since they're so security conscious, all the workstations have drives that are encrypted, have antivirus that runs every 4 hours, whether you're using the system or not, a couple other "intrusion detection" apps that also run, sometimes on overlapping schedules, sometimes when trying to patch the operating system... and for the bonus round: An account used for software installation that has full local admin to every workstation... and has a password that's the same as the account name.

        -_- Attaching one of those appliance timers to a switch to shut it off at predefined intervals seems so stupidly obvious, but when you realize how stupid the average person is, and then realize that the ones stupider than that work in HR and Accounting, you quickly conclude the same thing the rest of us in this industry have:

        Just drink your damn beer and try to drown out the stupid. Thinking about it will only depress you. Trying to do something about it will get you fired. Trust me... there is no faster way to get fired in IT than doing your job well... because you'll get noticed by all the incompetent asshats that HR and Accounting let in, and they'll form an alliance against you to get rid of you. And for the super jaded special bonus round... trying to get shit done will make you realize that the reason you can't get anything done is because everybody has silo'd themselves away with crucial documentation, settings, or knowledge, to assure themselves of continued employment. Start poking around, and they'll feel threatened. When they feel threatened, they'll find some way to go behind your back and make you look bad. Do this enough times and management will consider you an agitator and... ker-chop.

        If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.

        • Re: (Score:2, Insightful)

          by Anonymous Coward

          If you love computers at all, for the love of god, don't go into IT. It will shit in your soul.

          amen to that

        • You'd be surprised the kind of things that happen in your average large business thanks to HR and bean counters running the show and considering IT a cost center instead of an asset...

          Umm, IT actually IS a cost center in most companies. My company is a manufacturing firm and I run all our IT operations (among many other duties) and our customers do not pay us a dime for our IT. They pay us to deliver the products we make for them. IT falls in to the necessary/useful expense category. If you are not being paid for the IT then it is by definition a cost center. Lot's of necessary, useful and important things are cost centers - it's not a pejorative. The job of IT is to help the other

          • Well, when I say "cost center", I mean they look at IT resources the same way they look at paper towels for the bathroom... a necessary expense you should spend as little as possible on. Unfortunately, that's what they teach MBAs and the like in college. Doubly-unfortunate, they listen and never bother thinking on their own, or listening to people in the field.

            The difference between the cheapest solution and the second cheapest solution is usually the difference between "will work 4 out of 5 times" and "wil

      • by cusco (717999) <<moc.liamg> <ta> <ybxib.nairb>> on Friday June 07, 2013 @11:59PM (#43944021)
        That's fine, until the process control guys unplug from your nice managed port, run a cable across the floor and plug into a port that you're not actively managing. And they will do that. If you don't think so then you haven't worked in that type of environment.
        • Agree. They will do this. Seen it everywhere. And if the run is too long for a cable, prepare for wireless.

          ~Sticky

        • by mlts (1038732) *

          That, or you might find an el cheapo four port switch between a production machine and its normal port where there wasn't one before, so unless you set the max MAC addies to one as a matter of daily business, you may have an ugly surprise.

    • by Nikker (749551)
      I agree there are too many problems that this could cause (putting the machines on a 'timer') than they would benifit from security. If any company is having mission critical hardware / software handled through the internet you must have engineers or at least senior support staff present to make sure all goes well, after all when are you going to ensure the equipment is running properly, Monday morning 8am?

      Have the connection closed with a metal case and a key, when the time comes the company doing the
  • by litewoheat (179018) on Friday June 07, 2013 @10:26PM (#43943555)
    You could stick an old machine with 2 NICs running Linux in the middle running a simple proxy written in nodeJS. Since you've written the proxy you can write all the rules you want. With node it would be incredibly easy.
  • by srbell (164773) on Friday June 07, 2013 @10:26PM (#43943559)

    I'm a DCS system admin at an oil refinery. We keep the DCS and business lans totally separated, and that directive is driven from the top down. If anyone asks for remote access we just let them know that's NOT going to happen - end of story! It can be a pain getting files from one network to the other (patches, etc.) but certainly worth the effort.

    • Thank You (Score:4, Insightful)

      by SuperCharlie (1068072) on Saturday June 08, 2013 @02:16AM (#43944395)
      We hear of so many idiots with critical infrastructure connected to the Internet that I felt it my duty to single this post out as #outbreakofcommonsense and say thank you for fighting the good , non-moron fight

      Hats off to you sir (or madam as the case may be.)
  • In this case it would have been a futile effort, since they would have called to have the connection made and messed up the upgrade anyhow! If you could trust the production people to do it, it could be as simple as unplugging an Ethernet cable and only plugging it in when it was absolutely necessary. Other wise a network switch can have individual ports turned off remotely. Not an IT that's my son, but this is what I found on a quick look.
  • your are describing managers who are incompetent and not qualified to run their business.

    you have done what you can do. you arent the company president.

    this is like dealing with a durg addict. you cant save them. you cant change them. they have to want to change, and they dont want to.

    let them fail. let them go bankrupt.

    make sure you cover your ass. keep documentation of their stupidity and your warnings.

    and keep your resume updated.

  • by Mike Hicks (244) <hick0088@tc.umn.edu> on Friday June 07, 2013 @10:28PM (#43943583) Homepage Journal

    If this system is using an Ethernet connection, just get a Linux or *BSD box running with bridged Ethernet interfaces or pay for a decent smart switch. Heck, you could probably do it in Windows -- that supports bridged interfaces too.

    Simply disable the interface connected to the device you want to protect whenever you don't want outside access. With a Linux/*BSD box, this could be accomplished with simple scripts. You'd probably have to write up a simple manual procedure to do it with a switch or Windows box.

  • I don't know what type of router you have but many do have scheduling capabilities. Actually publishing information like brand and model of router would be pretty dumb.
    My first step would be to contact your router manufacturer and if necessary get one that has that capability. You could even put all of your manufacturing equipment behind one unit on it's own segment of the network with limited access from outside, assuming that you really need network access at all.
    Unplugging from the network is an option

    • I don't know what type of router you have but many do have scheduling capabilities.

      BINGO.

      A freaking consumer-grade Netgear router does this!!

      (I can verify first-hand, as I just bought one about a week and a half ago.)

  • Assuming you have managed switches a simple crontab entry pointing to a shell script can open a connection to the switch an admin down the port that its plugged into. If you want to get really fancy you can have the outbound traffic going via a transparent squid proxy / iptables so you can tell when the port is in use, and keep logs of the connection state.

    You can also go with a non-NAT firewall (bridge mode), which will block incoming connections while the device / people on the inside wont know anything i
  • by Anonymous Coward on Friday June 07, 2013 @10:35PM (#43943625)

    > As we know, process control workers and vendors are woefully inept/uneducated about IT systems and risks

    If you're going to call someone inept, you better make sure you're not, especially if its your own FUCKING FIELD.

  • Oh the irony (Score:5, Insightful)

    by Anonymous Coward on Friday June 07, 2013 @10:38PM (#43943649)

    How on Earth do YOU get to make fun of other employees at that company? I can think of at least a couple of filtering methods more elegant than a freaking christmas tree timer and I'm not even in IT. If all departments' staff quality is the same as IT I just hope that the "large industrial equipment" is not something that can affect other people.

    Filtering access on a per-request basis is one thing, and I see how that's critical and can't think why you haven't implemented this already. Filtering access on a per-timer basis is the WORST WORST WORST idea ever. If I could make that any more caps locked I would. There are SO many things that can go wrong with a blind timer-based disconnection that I won't even bother to list them all, I will just paint the simplest of pictures in a newspaper title: "Incomplete update to a CNC machine leads to hands being sawn off".

    Do yourself a favor and change jobs.

  • Password control? You could implement a policy where all remote connections must be sanctioned by someone on your side and disable/change passwords fatter the remote user notifies you their work is done or after a predetermined amount of time. Bear in mind you should read your support contract before you try this -you may find nasty penalties or be in breach.
  • Either use managed switches to separate the LAN's and lock down the ports to only allow certain mac addresses on the PCS VLAN, or create another LAN with dumb switches. Add a VPN box that is connected to both networks, make it the only allowed method of connecting remotely to the PCS LAN, give the vendor a VPN account, and only enable the account temporarily when it has been approved.

  • by Jaktar (975138) on Friday June 07, 2013 @10:59PM (#43943767)

    I think the OP is missing something.

    I do process control. It's not manufacturing, but that part is irrelevant anyways. The issue at hand is that process control has shifted to control systems that are networked. There are options that don't use ethernet/ethernetIP, but they're increasingly going the way of the Dodo.

    We're in a strange time when control systems are increasingly being networked, and the guys that used to do control/automation (and used to do it with relay/hydraulic/pneumatic) don't have the necessary training to integrate the systems correctly. Most IT people don't understand how control systems work and the implications of changing network configurations.

    The way forward is to merge IT and process control. Unfortunately, that's easier said than done.

    • by omglolbah (731566)

      Most know, some fight to get security put in place...

      But management in-house and managers at the customer tend to view security as a needless expense. Mostly because they have 'a firewall' (non specific...) and believe that one layer of security is plenty. Especially since 'the vendor promised it was 100% secure'.... sigh

      Oil rig PCS network 'secure plant network' that goes onshore office network internet.

      Since the firewalls 'are secure' the management think there is no even theoretical way for anyone to

    • by i.r.id10t (595143)

      The IT side and process control side should each pick someone to go and learn how the other side works for a while. A quick tutorial on basic networking concepts, a second on network security and infrastructure set up, and then actually being the PFY or Intern for a few days. The IT designate would go to process control and do similar.

      Basically, each side has to have someone who at least has an idea as to what goes on on the other side.

    • by cusco (717999)
      Security systems are in much the same boat. When I got into the physical security field most place's access control system ran off an old junk PC shoved under a janitor's desk, with serial cables running to the controllers. Camera systems were all analog and most DVRs were not on the network. It's been a really big change for that industry, and most of the VARs don't have the training to make the change.
  • by Joe_Dragon (2206452) on Friday June 07, 2013 @11:00PM (#43943773)

    check your contracts before doing any thing you may be on the hook for the full cost of that large industrial equipment after you break the contract

  • You are looking for time-restricted firewall rules.

    You can roll them yourself (Linux, Free-BSD), by just having two sets offirewall rules, and switching to the restricted set after the time expires. If you re-inilialize or don't use connection tracking, existing connections get cut. Reloading a firewall rule-set does not cut connections if it does not take too long). You can also isolate this in a specific sub-chain, and then just reload that one to enable or disable the specific connection. That way you ha

  • health Care systems are some times like this in where out side suppliers and vendors have control over systems and some times they don't install updates / say you can't over firewall this system.

  • by rusty0101 (565565) on Friday June 07, 2013 @11:12PM (#43943827) Homepage Journal

    ...is a post incident review with support people involved, and their management teams, along with directors and executive involvement to identify what the problem was that caused the business to be inoperative for the duration of the incident, what policies and procedures need to be followed going forwards, and so on. Once policies are established, solutions that support those policies can be implemented.

    As an example for your situation, since a vendor was involved in an upgrade, that should have been part of a scheduled change. The change should be documented ahead of time as to what is being done, what systems are going to be touched, and who the responsible parties both within the company and external to the company are for that change. Included in the documentation should be the fallback plan for dealing with issues that crop up during and after the change, within an appropriate test window that is included in the change window, as well as clearly defined backout procedures. "fix and fall forward" or equivalent statements are not, and should not be, considered acceptable plans. Wherever possible you want to have documentation attached that the procedures involved have been tested in a suitable test environment. (This may not be possible in situations where a test environment would cost as much to prepare as the production environment.)

    As far as limiting remote access, as others have pointed out, such limits are trivial based on what type of remote access is in place, and what policies are established. At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication, be specific to the time when those changes are scheduled to happen, and should not allow similar access to devices or types of devices not involved in the change.

    • by Animats (122034)

      ...is a post incident review with support people involved, and their management teams, along with directors and executive involvement

      There are some useful training materials from Homeland Security on this. See the National Infrastructure Protection Plan [dhs.gov] One of the key points there is to focus not only on prevention, but fast recovery. You may want to have spare control units on site which can be swapped in if the main ones are corrupted, for example.

    • by Z34107 (925136) on Saturday June 08, 2013 @12:42AM (#43944169)

      At the very least account authorizations required for performing changes on production devices should require someone in house approve that authentication

      I work for the "vendor" side of the equation. If we make any changes to a customer system outside this explicit, per-case authorization, we lose any limitations on liability. If we caused a downtime like in TFA, we'd be liable for up to $infinity in lost revenue, overtime, and other damages.

      As Rusty says, OP absolutely, positively needs to have a change control process with teeth if it's not followed. If his organization's support contract lets the vendor off the hook for this, they got taken for an expensive ride.

  • The best solution is to use this event as a jumping point into securing it right... No matter what technical solution you come up with, the weakest link are the people. Education, some firings, and getting a better vendor are the real next step. Remote access can be a marvellous tool to getting problems straightened out without flying people in, but it sounds like these are the kind of people you wouldn't let walk unescorted in the plant...
  • Plug the switch into a web controlled power switch:
    http://www.digital-loggers.com/lpc.html [digital-loggers.com]

    Eight power jacks that can be independently controlled over your network. You can control access to the entire device or individual sockets with multiple users and passwords, and they have built-in scripting functionality that shut off sockets based on the time, power-cycle if a repeating ping test ever fails to get a response, and other options I haven't bothered to look into. A real party. I think they're about $
  • by SuperBanana (662181) on Friday June 07, 2013 @11:40PM (#43943949)

    The supplier, with no notice, remotely connected to the process control system and completely botched an update to their system. We are down and the vendor is inept and not likely to have us back to 100% for a few days.

    This isn't a technology problem.

    Through their incompetence, they caused damages. Collect your evidence, hire a lawyer, and make demands. If they refuse to pay, sue them.

    Watch how fast they start caring about doing remote upgrades more carefully, competently, and with customer involvement. The only thing companies collectively care about is making money. At the very least, you'll cause their liability insurance rates to go up.

  • Cut a network cable and break the wires out, one to each port of one of these:

    http://www.sainsmart.com/8-channel-dc-5v-relay-module-for-arduino-pic-arm-dsp-avr-msp430-ttl-logic.html?___store=en&___store=en [sainsmart.com]

    Then, connect it to an ATtiny - ATtiny85 should be fine; it's only a couple of bucks - and program it to go on and off as you wish. Side benefit: it runs on batteries in case the power goes off.

    If you submit the idea to Make, I get credit.
    • No. What happens to equipment or people if lightning strikes nearby, or if a major pump shorts out? Will it transmit the current into the process switches, causing a larger issue? Will it electrocute someone nearby? Questions like these need to be answered before tossing equipment into an industrial environment.

      Neat idea, needs more than just an ATtiny. It was good though that you picked a relay that requires power from the ATtiny to turn on, I've seen other guys accidentally set stuff to fail open when t

  • by the_B0fh (208483) on Friday June 07, 2013 @11:43PM (#43943973) Homepage

    All Internet connections must cross a Firewall. Disable inbound connections, done.

    Seriously, this is a question?

    • Some vendors require this kind of remote access during warranty period of their equipment. Basically, the equipment doesn't belong to the client fully until it has met all requirements in the contract. Typically, this is 3 months to a year of service under operating conditions specified. So, what do you do when your contract requires you to keep a door open for the vendor, or otherwise absorb the risk of a ~1-5 million dollar job not being supported by them? Additionally, the guys allowing the vendors are
      • by the_B0fh (208483)

        You have never heard of change management?

        When access is needed, and authorized/approved, the firewall team opens it.
        When the work is done, the firewall team removes the permission.

        Is that really a difficult concept?

        • In IT, it's a very easy concept. Process control and industrial control systems is another matter entirely. They don't have a firewall team, or an IT staff, or a network admin, or a Windows Domain Architect, or any of that stuff. They don't have 4 days to wait for a change control board to approve access, because they usually need the vendor to fix crap immediately, or lose a few hundred thousand dollars in lost product.

          They have Steve, who has been at the plant since God stopped by for tacos. Steve knows

          • by the_B0fh (208483)

            If they understand the concept of guards at the gate checking people and calling in to make sure visitors were expected, they understand the concept of firewalls.

            It doesn't matter if these guys do not have any idea what good IT practices are. If it's multimillion dollars, this is a corporate function. All they have to do is call this number, and say "open sesame". And when done, say "close sesame".

            The corporate IT firewall guys can do the magic.

  • You wouldn't have an osm to worry about.

    ~Sticky

  • Configure your firewall rules so that access to these systems has to come from the internal network or through a VPN.. Create a VPN account for the vendor but leave it disabled. If there is a legitimate need to access the system they can submit a request through your internal change control process (or have an internal contact submit the request on their behalf) and pending approval the VPN account can be enabled when the work is to begin and disabled again once the work is complete. In fact if you are subj
  • Not your job. (Score:5, Insightful)

    by goodmanj (234846) on Saturday June 08, 2013 @12:04AM (#43944047)

    This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.

    Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.

    By all means give people support in doing their jobs, but don't do their jobs for them.

    • This isn't my field, but I think you should do nothing. IT's job is to provide network access. Process Control's job is to keep the machinery running, and if they fail to do so despite your warnings, it's their ass on the line.

      Yes, "not my problem" is a classic way to make a workplace awful, but consider this: if Process Control can't get a software update to their machinery because you've blocked it, and something bad happens (worst-case scenario, a machine kills someone), then it's *your* ass on the line.

      By all means give people support in doing their jobs, but don't do their jobs for them.

      I agree. IT ISN'T YOUR JOB.

      It sounds like your Process control department has outsourced support to the system to this particular company. They are likely monitoring it and this requires always-on access. When a problem does occur, it's up to management to enforce any vendor SLAs that were negotiated and any relevant penalties.

      I'm a network engineer for a large company and we also have a separate manufacturing group that has it's own IT. The best that you can do is make sure that there is a firewall bet

    • Yup. I'm a controls engineer, and 1. I don't expect IT to either fully understand the nuances distributed controls architecture, or 2. do my job for me. I don't fix accounting network gear, they don't fix my DCS. Knowledge of how to manage TCP/IP networks on the factory floor is Engineering's job, not IT's.

      That said, it sounds like OP's engineering manager is a fucking idiot that needs to be canned, along with the staff that tolerated it. This is how refinery explosions happen. This sort of laziness and lac

    • The I told you so and not our problem method is not what's going to get the right result here. What you've done is passed the blame but not fixed the underlying cause of the problem which is a fundamentally flawed approach to dealing with your systems.

      Both the process control team and IT have experience that the other team doesn't have. What is needed is for both teams to sit down and nut out what a site wide acceptable policy for ALL gear is and then have the upper level of management sign off on the polic

  • by Loudog (9867) <loudog&doghaus,org> on Saturday June 08, 2013 @01:19AM (#43944273) Homepage

    Ignore the haters, they don't understand the politics for this. I used to design industrial Ethernet networks for a large vendor, and we spent quite a bit of time pointing out to customers how dangerous the direct lines were. However, IT departments have very little say over manufacturing networks. This isn't always a bad thing (see the many IT/help desk horror stories). Because the remote access is often required as part of the maintenance contract, offer to partner with manufacturing to install a small firewall with access filters that are controlled by IT, but set (requested) by manufacturing.

    A small Cisco ASA, Juniper SRX or its like will do the job nicely, and can shield you from hack attempts along that access path.

  • This is a problem with the organization of your business and cannot be fixed with a technical solution.

    "the main issue is that they were able to do this at all, but reality is that IT gets overridden by the Process Control department in a manufacturing business. They were warned about this"

    You need to raise the risks to management and if they choose to override your recommendation then the problem is theirs, not yours.

  • You said they need remote access and botched the upgrade. There is no technological solution for this. If your management insists an incompetent vendor have such access and thinks you can do something about this result then the solution is to find another job. If you aren't being told to fix it, realize that it's not your problem.

    You can't fix stupid; you either have to accept it or escape it.

    • by roc97007 (608802)

      But management is usually gauged, by their superiors or the board of directors, on results. A major production outage is not a good result. This can be used as leverage to get the process changed, if one has the guts to pursue it.

  • by roc97007 (608802) on Saturday June 08, 2013 @02:39AM (#43944469) Journal

    > but reality is that IT gets overridden by the Process Control department in a manufacturing business

    It happens in a lot of industries. We're forever chasing vendors who think it's ok to pull our systems out from under us to apply updates, sometimes (thankfully rarely) bricking the systems keeping us down until they can make physical repairs.

    I don't think there's a surefire technical solution. We disallow access from outside directly to our hardware via our firewall (the best solution -- don't think christmas tree timer, think firewall or switch controls) but since the outsourcing, our firewall is itself under management from an outside group (albeit a different one) and they don't seem to know what they're doing, except to call an operator to press the reset button when a problem is reported.

    But the point is, the problem is a social one, not a technical one. I know you haven't had good results so far, but this needs to be fought in management, not in technology. A major production outage gives you fuel -- get riled up, and go talk to some people. Make it plain that the next time the vendor makes any change at all without first approval from a cross-department board, will be the last act that particular vendor does in your company. Put some teeth in your service contract. Hop to it. Your company is at stake.

  • The device (Score:5, Insightful)

    by gnu-sucks (561404) on Saturday June 08, 2013 @02:46AM (#43944489) Journal

    The device is called a "firewall" and is set up by an "IT Professional"

    You tell the IT guys when (or if) you want that company to be able to connect in remotely. That's it.

  • Who is more inept? The inept or the inept who hired himt? ~Obiwan (while working IT)
  • -keyboar battt dead

  • This Ask Slashdot has to be in the top 10 worst Ask Slashdots...
  • "Is there a device to automatically disconnect network or otherwise time limit a physical connection to a network?"

    Yep, it's called a router.

  • Seems to be something that needs to, at least in part, be governed by a contract with the supplier. You might find yourself the scapegoat if off your own back you buile q Wallace & Gromit inspired steam-powered Kill'O'Network contraption.

  • ELEGANT! ELEGANT!

    Sheesh. IPTables, Cisco timeout functions...no imagination whatsoever.

    Write an app for a dedicated iPhone (hooked to a power adapter) that is interfaced with an RJ45 switch (low-voltage control circuit for a small relay in place of the usual rotary switch) for the ethernet cable , so that the vendor has to call that iPhone and maintain a connection to keep the ethernet connection active. Limit the phone number sharing to vendors only.

     

  • "Something with two network jacks on it that disconnects the port after a set time"

    No wonder some people have a hard time finding jobs, since a lot seem to be taken by cheap labor with unrelated or irrelevant knowledge or workforce repurposed from other departments to cut the costs of hiring someone who has the proper knowledge. I am not in IT, never was, never will, yet even I know of the device the poster seeks with multiple jacks and connection handling functions, which are magic boxes brought by blue
  • by Tom (822)

    Why do you want to disconnect after a set time interval? Your time will never be correct. Sometimes, legit maintainance might take longer (e.g. if they're monitoring the system to trace a problem) and often it will be much shorter.

    Without knowing the details, what you need is a point inbetween that IT Security controls, and the procedure that says enabling remote access requires form 123a filled out. Which would be a simple paper saying "please enable remote access for vendor X on (datetime) until (datetime

  • by Virtucon (127420) on Saturday June 08, 2013 @07:09AM (#43945309)

    If somebody wants remote access to manage "their" system, these are a few of the things you should insist on:

        a) Explicit Contract statements describing the methods of access permitted, when it's permitted and by whom.
        b) Contract must spell out the type of testing or diagnostic work to be performed by vendor technicians and who on your company side will pair up with them to validate. The buddy system is to be used at all times when the vendor is gerfinkerpoken mit das machinen.
        c) Managed Access, only allow them in on incidents of failure for diagnostic work or for upgrades again, pairing somebody inside your org up with them during the access window. No Carte Blanche access at all, no VPN tokens and no dial up. All access must be over a minimum of TLS 1.1 links, TLS 1.2 is preferred.
        d) Penalties in the contract for fucking up. Make it so nasty that if the fuck up they'll pay through the nose if they take you down. Sorry it has to be this way.
        e) Specify that you require test results and approval of the test results (including how they tested) before any upgrades. Also provide a test infrastructure or subsystem to allow the vendor to deploy to first to verify that what they're saying actually works. Just because they've done testing doesn't necessarily mean it will work with your hardware and your environment.
        f) All workers from the vendor doing the work must be based in your country and be primary to the vendor. No third parties are to have access to your infrastructure or systems. Don't let subcontractors who don't know your systems and processes in.
        g) During upgrades or problem sessions a hot call is established to let key stakeholders in your company know what's going on and to provide progress updates on rediation or the success/failure of the situation. Keep your management in the loop and make sure they're aware of the scope of any changes.
        h) Backout plans must be provided in case of failures of any changes. I realize this may not be possible with some PLC/Process systems but that should also be a consideration when purchasing these kinds of systems.
        i) Maintain air gaps at all times between PLC equipment and any network infrastructure that has access to the Internet or Corporate Intranet. No connections to networks with "office" applications or information flow should be allowed anywhere near your process control networks. The exception to this is when the vendor is troubleshooting or upgrading systems or obtaining log data in accordance with the rest of this.

We warn the reader in advance that the proof presented here depends on a clever but highly unmotivated trick. -- Howard Anton, "Elementary Linear Algebra"

Working...