Nuclear Plant Operators… GASP…. Surfing the internet???
January 12th, 2012
|
| Share |
Okay, I admit it. I’ve been at work in a circumstance where I should have been writing code or responding to e-mails and I may have hit up Facebook or Google News. Sometimes I had a half-assed excuse to it, like that the weather was bad and I needed to know if there were any impending weather emergencies that might force the business to close early. I might also say justify my Facebook surfing as “exploring the possibilities of social marketing.” The fact of the matter is that I was slacking a little from time to time. Who amongst us hasn’t?
But uh oh… it seems nuclear plant operators may have surfed the net
NRC: Nuclear technicians surfed web on the job
Nine technicians responsible for monitoring operations at a Louisiana nuclear power plant spent on-duty time surfing the Internet — visiting websites that included news, sports, fishing and retirement information — jeopardizing the safety of the plant, federal regulators say.The Nuclear Regulatory Commission disclosed the web-surfing activities Monday in a letter that proposes a $140,000 fine against the River Bend nuclear power station, 24 miles northwest of Baton Rouge.
No pornography sites were accessed, the Nuclear Regulatory Commission said. And importantly, the NRC said, the computer use did not present an avenue for hackers to gain access to reactor control systems, a modern-day fear at industrial plants.
But the NRC said the web-surfing control room operators were directly responsible for monitoring the reactor and other plant systems, and that their actions violated plant procedures requiring operators to remain attentive and focused on their work.
According to an NRC investigation, nine operators “deliberately violated” the safety procedures by surfing the web between January and April of 2010. Three of the nine did so with such frequency and duration that they are being issued “severity level three enforcement violations.” (Severity level one represents the greatest significant violation and severity level four is the lowest.) The remaining six operators will receive severity level four violations.
The operators were not named by the NRC.
An NRC spokesman said the proposed fine for web surfing is the only such action for web surfing in memory, and may be the only such action in the history of the agency.
In a notice to Entergy Operations Inc., operators of the River Bend Station, the NRC said that it appears that operators “remained attentive to reactor operations, indications, and alarms” while surfing the Internet.
“However, because most of the operators involved knew and understood” the prohibitions on Internet access, they exhibited “deliberate misconduct” and engaged in “hundreds of instances” of accessing the Internet from the “at-the-controls” area of the control room.
Score one for ridiculously reporting.
No, there was never a safety risk. While I don’t know exactly what the operators were assigned to do or how the systems operated here, all indications are that they were simply passing some time by surfing the net when they didn’t have any need to directly interact with the controls. Nuclear reactors certainly do not require continuous second by second human input nor do they need to have a reactor operator spending hours blankly staring at the dials that usually don’t change. Granted, all indicators are checked frequently, as they should be, but that was never interrupted.
It seems that in this case the operators were doing something many of us have: using company computers with internet access for personal surfing. Companies don’t like this, of course, because it tends to encourage employees to spend their time non-productively. If not for the internet, the operators might be more prone to doing something more useful for the company during the time they spend babysitting the control room. It’s like anything else, where the operator is primarily there for contingencies or if problems arise.
Still, this really just isn’t a news story. The workers never left their posts and they were ready to respond to any incident. That’s the important thing. I guess in the future they’ll have to go back to old fashioned paper crossword puzzles and magazines.
This entry was posted on Thursday, January 12th, 2012 at 8:06 pm and is filed under Bad Science, Nuclear, media. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
View blog reactions
January 13th, 2012 at 1:05 am
Does anyone know to what extent a nuclear control room actually needs to have people constantly turning knobs and flipping switches and to what extent they basically just babysit the equipment and watch for it to do something that needs attention?
Quote Comment
January 13th, 2012 at 4:03 am
Ryan said:
The vast majority of the time it’s a case of just being ready in case something happens. In that respect, it’s a bit like being a firefighter – the best case scenario for everyone is that your entire time spent at work is just waiting around with nothing to do.
Perhaps a better analogy is that of a freight train driver: starting and stopping properly will require some proper attention, there will be intersections and junctions every so often where you’ll have to keep and eye out to make sure everything is okay, but a big chunk of the time is just spent with one eye on the dials and another on the scenery flashing by.
Having said that, there’s still a certain amount of activity floating around the average control room. There’s regular testing and maintenance stuff to do which will mean being called up by maintenance teams (who will, in all likelihood, have come in to see you earlier to cross-brief) to flick some switches around the room. Then there’s paperwork to keep up on, mainly just ensuring that the dataloggers are all okay and still drawing lines on bits of paper.
Given the immense complexity, policies of redundancy and safety culture in nuclear power, in my experience, control rooms are rarely silent places. I’m not sure I’ve ever been in one without at least one alarm going off. Don’t read too much into that: nuclear plants have alarms for everything, and 90% of them are no more important than the oil temperature light on your car coming on when you first start it up.
One I can recall from memory was just an electronic twinkling noise, like a watch alarm. When I asked, I was told that it was that one of the three injection pumps was offline – for standard maintenance – but that only one was ever required at a time. The alarm was just a way of making it hard to forget, not important enough to get a proper alarm or a proper flashing warning, nor minor enough to be just a twinkling light among dozens on the wall displays.
Quote Comment
January 13th, 2012 at 4:16 am
I work in the nuclear power industry and worked on shift in the 1980’s when the closest thing to an internet was bulletin board systems that computer geeks would connect to over their home phone lines using 2400 baud modems. Beyond hourly readings that needed to be taken in the control room and twice shiftly rounds to be taken by equipment operators in the plant, backshifts could get pretty monotonous. A lot of time was spent by the operators gossiping and telling sea stories (most plant operators were former nuclear sub operators). Crews work rotating shifts that can wreak hell on your circadian rythms if you didn’t work at it by staying up a couple hours later each day so that by the time you transition from day shift to swing shift or swing shift to the midnight shift your system was already acclimated. Even then fending off boredom and staying alert was a challenge. I don’t know how many times I spent time rereading procedures or system descriptions to get ready for the next training cycle only to find myself having stared at the same page for 10 minutes without having any idea what it said.
Society evolves. What people could get away with in the past, like businessmen pinching an airline stewardess’ butt on a cross-country flight or telling ethnic jokes in public are not tolerated today. By the same token, the level of professionalism the nuclear industry expects of its operators has also increased over time with things that were merely frowned on in the past, like reading a newspaper in the control room, becoming serious breaches of conduct today and other things that were acceptable behavior being frowned on now. Back in the day it was common for operators to report for back shifts wearing Harley Davidson tee shirts and dirty jeans. Now there are dress codes, as seen in the photo.
As I see it, the offense wasn’t so much that the operators surfed the web but that management didn’t get in front of the problem and reinforce industry standards for conduct when it first became aware of the issue. There is a lot of peer pressure in the industry. If one plant does something that generates negative publicity, whether justified or blown out of proportion, it makes all nuclear plants look bad. The industry isn’t that big; the utility chief nuclear officers and site vice presidents know each other and this sort of thing is an embarassment to them. Nobody wants to look bad in front of their peers and I would not be surprised if policy changes are put in place that raise the bar for everyone else to follow.
Another aspect to consider is that cybersecurity is now a very big deal, especially at nuclear plants. Over Christmas my brother and I spent a good deal of time trying to save our teenage niece’s laptop. She had downloaded a “Free 3-D screensaver!” and installed a pink mouse cursor that sparkled, among other things that had loaded all manner of malware. Not that anyone would expect to download a stuxnet virus from visiting Cabella’s web site to window shop for hunting rifles, but once one behavior becomes tolerated the natural tendency is to stretch the envelope.
Quote Comment
January 13th, 2012 at 4:48 am
I should add that most nuclear utility IT departments lock down their internet access pretty tight. All social media and many commercial sites are banned. I can’t download or watch YouTube videos from work, for example. I doubt I could get to the Bass Pro site if I tried. Our IT can get overzealous and I have on occassion had to go to them to request access to legitimate sites like certain NRC portals.
Quote Comment
January 13th, 2012 at 5:34 am
The only hazard I can think of is for computers somehow connected to important apparatus of the plant.
Given what Stuxnet was capable of I really hope that those computer don’t use windows and are isolated as much as possible from the rest of the world.
Other than that, the worst case scenario could be that the “office” computers being knocked out from some virus (especially if windows is used) impeding some not critical activities in the plant.
Quote Comment
January 13th, 2012 at 6:37 am
There was a case of a nuclear plant having its computers taken down by a worm (the plant was shut down at the time, and the operators were able to get by on the analogue backups).
See http://www.securityfocus.com/news/6767
Quote Comment
January 13th, 2012 at 7:04 am
Massimo C said:
Most plants were built in the 1970s and 1980s before programmable logic controllers became popular and the logic systems built in the control boards for automatically shutting down the plant or starting safety systems are controlled using electromechanical relays that are hard wired, not programmable. PLCs are being used more and more during system upgrades for “balance of plant” controls like feedwater heater drain valves. Messing with those could cause the plant to shutdown, but would not affect safety systems. The old fashioned “steam guages” and alarms you see in the picture are safe from tampering. There was an instance of a vendor plugging in a laptop that he had also done personal surfing on to a plant’s process computer to perform diagnostics and unknowingly uploaded malware. The process computer, as I recall, is normally pretty well isolated from the internet and provides computer indication, calculates certain safety margins and does datalogging. I have no doubt the system the control room operators were using were physically isolated from the plants control systems but the nuclear industry operates on the concept of building in multiple layers of defense and breaching any one of them is taken seriously. The NRC and industry have put a lot of thought into cybersecurity guidelines. It is a big deal in the industry. The fines the NRC doled out in this case are evidence of how seriously it takes even minor lapses that had no concequences.
Quote Comment
January 13th, 2012 at 9:11 am
@Blubba Yeah I expected that the security for those facilities is taken very seriously.
As a potential threat I was thinkng more on subltle changes not big things that can instantly trip the shutdown of the plant.
TMI accident demonstrated that a small faulty (bad designed) indicator can misguide the operators and spiral the situation to the worse.
So an hypotethical malware can simply alter slightly the behaviour of some detectors or valves or something else to give false informations to operators, letting them to do the bad things unknowingly.
What Stuxnet did was simply changing the cycle of the centrifuges and masking some parameters to disguise itself.
Quote Comment
January 13th, 2012 at 9:19 am
Maybe allowing the control room operators to at least surf news and weather sites might not be such a bad idea.
Remember 9/11? That morning I was watching Good Morning America and my daughter and I saw the broadcast of the planes hitting the World Trade Center. When I saw the second plane hit on TV, I called my husband at the nuclear power plant where he was working to let him know what had happened. He relayed the information to management at the plant. Later there was much concern about how he had “inside information” of the attack.
While I understand the need to keep the plant personnel professional and alertly monitoring the plant condition, maybe keeping them a little less isolated from the rest of the world would actually be helpful.
Quote Comment
January 13th, 2012 at 9:40 am
Massimo C said:
Those kinds of subtle changes would indicate a targeted attack which is of course much harder to defend against (and would probably require that the computers running the plant systems not have any internet connection, not even a firewalled one to the internal network) since merely being less vulnerable than someone else or running unusual software won’t help (whoever developed Stuxnet (probably Mossad) really knew a lot about Iran’s uranium enrichment facilities).
Quote Comment
January 13th, 2012 at 3:49 pm
Blubba said:
True, although in practice that does not necessarily compromise the actual systems.
There is a difference between computers that are just there for administrative and office functions, such as sending and receiving e-mails, writing memos, logging time sheets and that kind of thing versus the ones that actually directly interface with the plant systems. These are often (and should be) separate systems.
Computers which are just used for those functions generally should be separate and have different levels of network access. Those computers should only have general purpose connections to the internet, e-mail, network printers and that kind of thing. The high security portion of the network should be as small as possible, avoiding unnecessary nodes that don’t need that connection to reduce potential entry points.
It is my preference that networks be separated such that updating the firmware or operating systems on system-control computers can never be done entirely remotely from the general purpose network.
Quote Comment
January 13th, 2012 at 3:53 pm
Blubba said:
I’d be curious about what their policy is about what you can bring in is. I mean, their network might well have sites blocked, but if the employees bring in their own 3 or 4G access cards, ipods, smartphones or iPads, then it really doesn’t matter. If they can get reception they can pretty much do as they please.
It’s not a nuclear industry problem per se, of course. I’ve heard of police playing video games on their own personal iPads when they should be paroling the streets.
Quote Comment
January 13th, 2012 at 6:33 pm
Anon said:
Let’s be clear, the article got it backwards. The analog system (the steam gauges you see on the control panels and the electromechanical relays behind them that drive the safety system logic are the primary system. It is the SPDS system (that could be pulled up on the computer monitors) that is the backup.
Quote Comment
January 13th, 2012 at 6:46 pm
Anon said:
But computers don’t run the plant systems. At least not the safety systems. As I’ve indicated before, they are analog (at least the older plants, I can’t say for sure about the later plants put on line). My understanding is some have retrofitted some of the non-safety systems with a few PLCs but the worst case scenario that I can see would be to cause the plant to trip offline, which would be more of an economic threat to the utility than a threat to the public.
Quote Comment
January 13th, 2012 at 7:45 pm
Blubba said:
Even if they use the computer more often?
Blubba said:
That’s pretty much the worst anyone could do even if they could directly control somethings unless they were able to keep control for days at a time which just isn’t likely to happen.
Though an economic threat to a utility isn’t something to dismiss out of hand (fooling the operators into writing off a perfectly good power plant is probably the worst someone could do).
Quote Comment
January 13th, 2012 at 11:21 pm
I know this will not be a popular take on the matter, but I do believe that when you are being paid to preform a certain task, in this case stand a watch in a control room of a nuclear plant, that is what you should be doing. I realize that in this case there was no real danger in what they were doing, and maybe I’m just oldschool, but don’t think that surfing the net during working hours is acceptable behavior.
Quote Comment
January 13th, 2012 at 11:30 pm
DV82XL said:
Oh, I agree that you do what you are supposed to do and what you have agreed to for work. If the rules are not to surf the net, then you don’t do it.
My point is more that this is not a newsworthy item and it’s not something people need to get all bothered or scared about. It’s an internal issue for the employer to deal with.
Quote Comment
January 14th, 2012 at 1:06 am
Anon said:
I am not sure I understand the question, but for every interpretation I can think that you mean by that the answer is “NO”. SPDS provides no control. Period. SPDS is sort of like the computer displays you might see at a modern refiner or brewery on TV programs like How Its Made that show simplified mimics of process flow and status, except they don’t provide touch screen control of equipment. If someone hacked into it and caused it to provide conflicting information the operator’s post-TMI trained response would be be to review all of the main control room ANALOG indications. Their assessment of the analog indications would trump whatever the computer display says. Their inclination would be “Damn SPDS is bad, initiate a work order to fix it and then notify the NRC and other users that the system is down, don’t rely on it until it is fixed” and simply ignore it until it is fixed. What SPDS is primarily used for it to provide a high level view of plant status that is accessible at the company’s emergency response facilities and to the NRC’s emergency facilities outside the control room. SPDS is a ccnvenience that would reduce control room operator burden by communicating basic plant parameters to those groups and allow them to see for themselves without having to ask. So hacking SPDS would not cause an event, but it would complicate the response for a TMI type accident. Let me repeat – SPDS would not cause a TMI but hacking it would complicate the communication of information to outside entities about the plant status.
Anon said:
See above.
Quote Comment
January 14th, 2012 at 1:11 am
DV82XL said:
You’ve nailed on the head the behavior the industry expects of control room operators, aside from any cybersecurity concerns.
Quote Comment
January 14th, 2012 at 1:18 am
drbuzz0 said:
The event sort of reminds me of the Northwest Airline pilots who got so wrapped up in their laptops and discussion about crew scheduling that they overshot the Minneapolis airport by 150 miles. Embarassing, no real risk to the passengers, but “newsworthy” by the standards of those who decide what “newsworthy” means.
Quote Comment
January 14th, 2012 at 1:56 am
Anon said:
To clarify my previous response, with respect to PLCs, as I understand how they have been used to date in US nuclear power plants, once a plant has tripped such devices would be pretty much removed from the equation. Even if they continued to control a feedwater header drain valve for days after it would be moot because it would physically be isolated from the systems that provide emergency control. If PLCs are allowed to be used in safety systems it will only be after the NRC issues guidance or approves industry proposals for standards governing their use and the controls that address the potential for a stuxnet attack.
Quote Comment
January 14th, 2012 at 12:17 pm
Lets not get too off topic, but there is certainly another big and fascinating subject regarding control system design theory and how you can make a system that is automated and yet safe from hacking or various software failures. It’s an increasingly important consideration as we move to more digital controls and automation.
Software and firmware driven systems offer a lot of flexibility and can be cheaper to design and implement than those that need to rely on entirely hardware based systems and we’re going to see a lot more of them. Of course, this does bring up very real concerns and the extent to which these kind of systems are used and what access is available is something that always needs to be examined.
Personally, I think that we are at the point where some of these systems are going a bit too far into introducing network-centric controls that are potentially susceptible to outside influences or complications. They may have unnecessary overhead. Very simple hardware systems, where a sensor triggers a relay and there’s no high level logic tend to be extremely reliable. But there are always tradeoffs.
When safety is the issue, I always prefer that there be some kind of final failsafe that is implemented through simple hardware that is passively safe and not by passable. Examples being things like fuses, thermal fuses, freeze-plugs, gravity-driven control rods, normally-closed valves that need power to remain open. Those kind of things are the ultimate line of defense. A freezeplug will always function, never have a problem with hackers, never fail due to a software glitch or an infinite loop. The same is true of fuses, gravity-driven feeds and such.
Quote Comment
January 16th, 2012 at 7:37 am
I wonder what the reaction would be if this story was not related to nuclear plant, but instead airline pilots? Ok, so not surfing the web. Perhaps – talking about what they did last holiday? Talking about their children’s education? Perhaps one of them is delving into the sports pages of the local newspaper once airborne and on route? Don’t tell them that does not happen!
When learning to fly (the small aircraft first) you are taught ‘it is what you don’t do with the controls which is more important than what you do with them’. I bet similar applies to a nuclear plant control room. Same applies to chemical plants, steel making plants and other similar industrial processes.
That is not to say ‘anyone’ should be surfing when working – but then that was addressed in the first few lines of the blog entry.
Quote Comment