Telecommunicators for Pinellas County’s new consolidated Regional 911 Center located on the third floor of the Public Safety Complex, 10750 Ulmerton Road in Largo, return to their workstations Sunday morning. This photo was taken during a tour of the facility July 31.
LARGO – Pinellas County’s new consolidated Regional 911 Center is up and running again after apparent lightning strikes shut down operations for approximately 45 minutes Friday morning.
The center reopened in the new Public Safety Complex at 9 a.m. Saturday, Bruce Moeller, interim chief of staff, said Sunday afternoon.
“Everything is back up and functioning. Everything is fine,” Moeller said.
He explained the sequence of events from Friday.
“A few minutes after 11 (a.m.), Sunstar, which is 2 miles west (of the Public Safety Complex), had a lightning strike,” he said.
Staff at Sunstar “heard and saw it as it happened,” Moeller said.
The lightning caused a small fire in the wall and the phone systems went down. Sunstar is located at 12490 Ulmerton Road.
A few minutes later, the 911 Center at the Public Safety Complex, 10750 Ulmerton Road in Largo, also took a strike. Shortly thereafter, there may have been a second strike. Dispatchers in the room, which has small windows, heard and saw a flash.
“They had static on their headphones, which indicated a technical or electrical problem in the system,” Moeller said. “No calls were coming through.”
Staff began procedures to move operations to the backup location. But when they called Sunstar to let them know that Verizon would be rerouting calls to that location, “They said ‘no, wait, wait, we just got hit’,” Moeller said.
Staff then notified St. Petersburg police that countywide 911 operations needed to move to its Emergency Communication Center. St. Petersburg began receiving and processing all emergency medical, fire and law enforcement calls for the county about 12:05 p.m. Staff from the county’s center was sent to assist.
“St. Petersburg’s Emergency Communications Center and the county’s consolidated Public Safety Complex are designed to provide redundant backup capabilities for continuity of operations in the event of disasters or major system failures, such as was encountered today,” according to a media release from St. Petersburg PD.
Moeller said the backup plan with St. Petersburg worked just as it was intended. He said the plan was similar to the mutual aid system for emergencies throughout the county.
Moeller said although St. Petersburg’s communication center used different software, the county’s software had already been downloaded into St. Petersburg’s system, so calls could immediately be rerouted and staff could go to work.
The estimated time that calls did not go through was 40 to 45 minutes, Moeller said.
“We want to thank St. Petersburg Police Department. They stepped right up and we were able to route the calls to them,” said Mary Burrell, public information manager with Pinellas County Communications.
The backup location at Sunstar was operational again by about 9 p.m. Friday and calls were rerouted to that location. By Saturday morning, the primary system was back online at the Public Safety Complex.
“Three lightning strikes are not all that unusual, but for them to occur and damage phone systems in two different places was not something expected,” Moeller said.
Staff is still investigating to find out exactly what happened. Computer cards in three servers were damaged and all had to be replaced. Moeller pointed out that the lightning had not affected radio operations.
“Radio remained fine the whole time,” he said.
Moeller said the building was equipped with surge protection and there is surge protection in the room where the computers are located as well as the cabinets that hold the computers.
“Somehow, for reasons we don’t understand yet, that wasn’t enough,” Moeller said. “We’re looking at it.”
Friday’s 911 outage was the first in more than 25 years, he said.
The new center opened July 23 with all new equipment, which is the same brand just new technology, Moeller said. He said the new technology is in use in hundreds of call centers throughout the country.
Moeller said county staff isn’t aware of any “adverse outcome” from the service outage.
“Clearly some calls would have been unanswered,” he said.
This is the second incident with the 911 phone system since the center opened less than a month ago. Moeller said late in the night Saturday, Aug. 15, through early morning Sunday, Aug. 16, some 911 callers received a busy signal after a router malfunctioned at Verizon’s central office in Tampa. He said most had redialed 911 and the calls had gone through.
“It was not a catastrophic failure,” he said. “Verizon identified the problem and replaced the router.”
He said eight people had informed county staff they had problems calling 911 and staff had learned of a few others having issues from other sources.
Moeller has called a meeting on Monday morning with Verizon, Duke Energy, 911 center staff, the county’s Business Technology staff and the contractor in charge of building the new Public Safety Complex.
“We’re going to do a thorough review,” he said. “We’re going to figure out what happened and make the system better, stronger,” he said.
Burrell suggested that the public might want to keep nonemergency numbers to local police and fire departments on hand just in case. Moeller agreed that was a good plan.
“Having nonemergency numbers available is something everyone should have for times there is no emergency (but they need assistance),” he said. “In this case, my guess would be that there were some who called (those numbers).”
He estimates it will take at least a week – maybe up to a month – before there are answers and potential solutions for Friday’s outage.
“We’re going to find out what happened and we’re going to move forward,” he said. “Every indication is that this was caused by a direct lightning strike.”