OLDSMAR — The city of Oldsmar became world renowned for all the wrong reasons after the North Pinellas community’s water treatment plant suffered a software breach over Super Bowl weekend.
The Feb. 5 hack, which investigators said involved an unknown party accessing the facility’s computer system and altering the chemical composition of the water supply, received international attention and shined a spotlight on the shortcomings of a critical component of the nation’s infrastructure system.
Officials said the breach attempted to raise the level of sodium hydroxide, commonly known as lye, in the water supply to dangerous levels. It was spotted by a plant worker, who notified a supervisor who subsequently called the Pinellas County Sheriff’s Office, leading some to praise the alert employee.
“I commend the vigilance of the staff to catch something like that,” said Josiah Cox, president and founder of Central States Water Resources, which operates more than 250 water treatment plants in five midwestern states. “Small systems actually a lot of times are harder to run than larger systems just because you don’t have the redundancies and larger staffs and the same resources. So, the fact that they were paying that close attention to what was going on was really awesome and shows how much they care.”
While the worker’s quick actions drew praise, the reason behind the breach, reportedly attributed to a combination of outdated software and lax screen-sharing practices, earned criticism from all corners of the globe. It has forced Oldsmar officials to reassess and upgrade the security measures at the facility.
“We have addressed the cyber-related deficiencies that were reported in several FBI bulletins,” City Manager Al Braithwaite said during a Feb. 16 City Council meeting. “There will be enhancements that I will recommend to council that we will make as a result of the investigation to ensure optimal cyber-security for all of Oldsmar’s critical assets.”
Mayor Eric Seidel thanked Braithwaite, Assistant City Manager Felicia Donnelly and Public Works staff “for all the hard work and extra effort that has gone in after the fact of the event,” and he pointed out that while the FBI “identified some deficiencies that the city manager mentioned, they are already being corrected or continue to be evaluated.
“This is a success story,” Seidel said. “Our monitoring protocols worked, our staff executed them to perfection, and there were other backups. I don’t think that can be said enough. We were breached. There’s no question. And we’ll make sure it doesn’t happen again.”
To that point, on March 2 the council unanimously approved an upgrade for the plant’s supervisory controls and data acquisition software that allows Clearwater firm McKim and Creed to replace the facility’s current computers and software. It will also install a “simplified yet robust program” that other local utilities, including Tampa Electric Co., use as well as install additional security, according to the agreement.
The work, which is being handled through a “piggyback agreement” using the same terms and conditions as the contract McKim and Creed has with the city of Tarpon Springs, is estimated to cost $64,948, according to city officials.
Seidel spoke about the impact the breach has had on the city and how they are working to prevent another such incident from happening.
“The events since the breach at the (Reverse Osmosis) Water Plant have been a test of quick actions, consistent communications and discipline for the city,” Seidel wrote in an email to Tampa Bay Newspapers. “The city leadership has taken quick corrective actions on the deficiencies discovered in our cyber security, and the plant has been disabled from any Internet connection since the event occurred and remains that way.”
Regarding the international attention the city received in the wake of the breach, Seidel wrote, “Dealing with the local, national and international media can be a challenge, but when one adds to the fact that we are not making new comments at the request of the FBI, Secret Service and the Sheriff’s department, makes it hard for the story to be accurately told to the public. The investigation remains very active currently and we are fully participating with the case.”
The mayor added, “Apparently, this event is not the first wake-up call, as there have been others who’ve experienced such breaches. But it is a reminder all critical infrastructures must be hardened to deal with today’s threats. Here in Oldsmar this is exactly what we are doing, and our citizens can rest assured it’s a priority.”
At a March 3 meeting of the Pinellas County Mayors Council, Seidel told his peers that media coverage was incorrect in stating this was the first time such a hack occurred. “I can tell you with certainty, it’s occurred dozens of times around the nation at water plants. That’s not a good thing, but it’s just to kind of say it’s a wake-up call, as it should be.”
He reiterated that monitoring protocols worked exactly as they are supposed to. “Somebody didn’t get lucky and see it happen,” Seidel said. “They saw what happened because that’s the protocol, and it didn’t make it to the secondary and third security levels because it didn’t get past the first one. So some of the information that went out was a little misleading.”
Seidel said he would share an unedited version of what occurred when FBI and Secret Service investigations conclude.