News and Product Updates
TC's JumboSwitch Helps Hurricane-Proof Jamaica Utility Network
11/08/2008Imagine being asked to "hurricane proof" your entire communication network, particularly your SCADA and Teleprotection networks, in less than a year.
Such was the situation that the management team at Jamaica Public Service Company Limited (JPS) faced in early 2007 according to Clive Wright, Manager, Electronic & Communication Systems.
"Predictive system resilience was a key issue" said Clive. "We wanted a 'multiple fault protection' network that would maximize our chances to survive hurricanes, including worst case access to substations and remote RTUs. We also wanted to upgrade our legacy communications infrastructure."
Before JPS deployed JumboSwitch into its network, it had already implemented a semi-secured Ethernet network for SCADA communications, direct serial fiber optic links for Teleprotection, and microwave links for some substations. Adding the JumboSwitch backbone network with its multi-service capability will integrate all this with Ethernet/IP technology and will help lower JPS employee stress levels when ominous billowing black clouds fill the eastern sky.
Jamaica has been hit hard by hurricanes in recent years. For example, the powerful Hurricane Ivan in 2005 and Hurricane Dean in 2007 caused extensive damage to JPS' transmission and distribution systems, resulting in widespread power outages. Customers in some sections of the island were without power for more than a month in each case due to the extent of the damaged caused by these hurricanes. In addition, a system failure in July 2006 resulted in an all-island blackout that resulted in customers being without power for up to eight hours. More than 550,000 industrial, commercial and residential customers were affected.
As a result, in early 2007 JPS commissioned a "SCADA Visibility Project" to prevent these devastating blackouts from re-occurring.
The SCADA Visibility Project had two main goals:
- To provide a redundant high-speed communications network that would ensure 100 percent SCADA visibility into all 69kV and 138kV substations
- To provision redundant teleprotection communications for the 138kV transmission lines that connect the 9 major substations.
Redundancy
JPS was looking for multiple levels of redundancy on its island-wide fiber optic network in addition to its existing microwave network. It wanted a backbone solution that would support T1, Ethernet, RS-232 and Voice for a variety of applications including SCADA, Teleprotection, Substation Telephones, Revenue Metering and I Distribution Automation. In addition, the new system needed to be fully managed, substation hardened, IEC 61850-3 & IEEE1613 compliant and operate over only one bi-directional fiber strand.
Up Park Substation in Kingston, Jamaica
"The Protection and Control Team (P&C) argued that SONET provided the performance necessary for redundant communications, even though the sticker price was significant," said Detorrie. "The Communications Team (ECS) argued that utilizing QoS and VLANs on a Gigabit Enabled backbone would deliver similar performance metrics as SONET without the high price tag.
"In addition, ECS argued that SONET is a legacy and proprietary solution that would lock the solution to a single vendor, a sore point for the aging communications infrastructure. ECS felt that any solution must be open and standards based to enable a wider pool of equipment and sources to be accessed. After lengthy discussions we decided to go with Ethernet." Once JPS decided on an Ethernet solution, it began an intense product search. "Ideally, we were looking for a multi-service platform solution that would collectively minimize the scale of un-deterministic elements in the entire system," said Clive. "In other words one fully redundant box, fiber optic, substation hardened that would support Ethernet, T1 & Serial communications." However, initial search results were less than encouraging, said Detorrie. "The capability of the redundant Ethernet Switches and other similar products available at that time fell short of what we really wanted," he said.
Enter JumboSwitch
Then, in late Spring of 2007 Detorrie ran across a preliminary announcement for a new product from TC Communications called the "JumboSwitch." He immediately recognized that the JumboSwitch would meet virtually all of their requirements including a sophisticated serial-over-IP technology that supports fast real time responses for tele-protections, a highly reliable TDM over-IP technology that supports bit error free T1 lines, enhanced SCADA communications and VoIP telephones for substations.
Metropolitan Kingston Ring: 11 JumboSwitches, 2 Fiber Strands for SCADA and Teleprotection Applications
Best of all, a long conversation with Nick Lee, TC's Technical Sales Manager, revealed that a complete Gigabit Ethernet JumboSwitch network could be implemented for less than one half the cost of a comparable SONET/SDH system.
The JumboSwitch also offered additional major benefits including centralized maintenance and asset management capabilities, (e.g. remote download and back-up), serial number and firmware version collections, temperature and power consumption real time readings, remote power cycling, redundant power supplies operating in 1+1 protection at all times and hot-swappable interface cards.
JPS JumboSwitch Network
The JumboSwitch portion of the JPS communication network currently transports Ethernet, Serial Data and T1 over two connected fiber optic redundant ring networks: one for the City of Kingston and one for the entire island of Jamaica.
JPS has currently installed 50 JumboSwitch nodes, 11 in its 2-fiber Metropolitan (Kingston) fi ber optic ring network and 39 nodes in its 1-fiber, bi-directional island-wide fi ber optic ring network. The Master JumboSwitch unit is located at the main JPS central office in Kingston. Four 4U high JumboSwitch units are installed at the Central Office, Up Park Camp, Rockport and West Kings House, 2U high units are installed at the other locations.
Depending on the location, JPS is currently using three different JumboSwitch interface cards: Ethernet, Turbo-Serial and T1. Ethernet traffic is primarily administration and security (cameras). On the Turbo-Serial card, one RS-232 port is used for SCADA and the other as a backup for teleprotection. The T1's connect to the Microwave network.
Turbo Serial Card
The Turbo-Serial interface card is used for real time critical communication between protective relays. This specially designed card enables, for example, a Relay connected to Port 1 on the RS-232 card at Up Park Camp substation to talk to the Relay connected to Port 1 at Rockport substation. When a fault or overload condition occurs, connected Fault Protection Switches are activated within 10 msec. As a result, the JumboSwitch network can replace or back up existing, direct optical *Mirrored-Bits RS-232 modem links.
Future plans could include adding RS-232/Fiber Optic telephones or VoIP Virtual PBX's at substations with JumboSwitches.
Because of the success of these tests, current plans include migrating some teleprotection circuits to the JumboSwitch GigE backbone.
JPS also successfully tested JumboSwitch T1 Interface Cards to be sure that they were compatible with the T1 microwave links. Typically, the T1 drop to insert delays needed to be less than about 6 m/sec.
Disaster Recovery
But, adding another higher layer network (JumboSwitch) with multi-service capability that could integrate their existing network with Ethernet/IP technology, provided JPS with an increased level of redundancy and disaster recovery capability that enhanced what both Clive and Detorrie like to refer to as "predictive system resilience."
"During a natural disaster such as a hurricane, we think more of 'predictive system resilience' rather than 'predictive system reliability' which is term more common to most U.S. Utilities," explained Detorrie. "In other words, how will the system predictably respond when a hurricane hits.
System Determinism
"By 'respond' the system should survive the event and restore normalcy to system operations in a predictable and deterministic manner. Immediately after the disaster passes, regardless of the impact, our immediate concern is 'How did things hold up?' The logical next step is to commence operations normalcy including telecommunication and Utility system operations."
JPS conducted simulation exercises in which it projected several disaster scenarios, including one scenario where it lost communication links to 8 out of 25 substations. In this worst case disaster scenario, the central office would be able to access substation RTUs by automatically switching between its microwave network and Jumboswitch network. (The JPS microwave device supports both T1 and Ethernet interfaces). In the event of an indirect hit (25% of island) by a Category 3 hurricane, JPS's goal is 99.9% communications system recovery within two hours. Recovery would likely take longer in the event of a direct hit (50% or more of island).
Because of the addition of JumboSwitch, the next time ominous billowing black clouds fi ll the eastern sky, JPS will feel even more confident that their system will survive and function in a predictable manner.
Conclusion
As JPS continues to implement and expand JumboSwitch capabilities in its network, Clive expressed satisfaction with their commitment to Ethernet. "The JPS network is strong evidence that IP-based technology can be used to deliver core services like SCADA and teleprotection to remote substations in addition to new and already successfully deployed services such as VoIP and video surveillance," said Clive. The Jamaica Public Service Company Limited (JPS) operates 47 substations and approximately 14,000 kilometers of transmission and distribution lines. It provides service to approximately 570,000 customers who use an average of 1,700 Mega Watts per day.