Menu
Ask any Texan who endured the power outages during a severe winter storm in February 2021 about the critical nature of the electrical grid, and you’re likely to receive an answer that is long on details of severe, even life-threatening hardship. We all depend on this critical energy source and infrastructure, yet operators are a long way from ensuring the protection of all components in electricity grids.
These conditions are not due entirely to a lack of regulations or industry standards. The Critical Infrastructure Protections of the North American Electric Reliability Corporation (NERC-CIP) are thorough and go a long way to ensure that OT systems in the electric grid are protected at the network and system level, and NERC has the capacity to hit violators with fines related to non-compliance.
However, NERC CIP regulations are not always applied to the distribution network, and do not currently provide sufficient guidance or high enough standards for securing embedded end devices, such as protection relays. While NERC CIP covers manufacturer’s patches and the enablement of security features, there are few mechanisms in place that would force a manufacturer to create a fundamentally more secure product. Many industry experts do recognize that embedded device security is below the maturity level it should be, given these devices’ importance to mission-critical systems, but this has not led to a concerted push for higher standards.
A number of factors contribute to this insecure stasis. Given the lack of regulation at this level and lack of demand from end users, device manufacturers are not incentivized to commit resources to developing more secure devices. While they may highlight specific security features or certifications their products earn, our research indicates that existing cyber certifications, while welcome, are not sufficient indicators of truly robust device security. The situation is rather like rating a car as “safe” without checking the condition of its tires, or some other essential component.
Regulations applied to different industries vary, but overall, there has been much more attention paid to the safety features of embedded devices rather than their security. While a “safety first” approach is reasonable, we argue that any discussion of embedded device safety that does not include a cybersecurity assessment is shortsighted at best, and a potential source of genuinely unsafe device operation.
To help bring awareness to the variability and gaps in safety-security standards for embedded devices, Red Balloon Security undertook a comprehensive analysis of three protection relays, each of which was released by a leading manufacturer and released after 2015.[1]
Each of these devices is currently available on international markets. Each highlights its cybersecurity features in its marketing specifications, including Safety and IEC63443 cybersecurity compliance, up to and including certification. And while this is far from an exhaustive analysis of the protection relay market, we believe our research is relevant to a general conversation about the electric grid devices (and also to embedded device security in many industries).
Protection relays play an essential role in the overall responsiveness and safe operations of electric grids. While far removed from the control room or IT systems, they are nonetheless accessible to determined attackers through several threat vectors (see our proof of concept for loading ransomware onto an embedded device).
The testing of these three devices included the following steps:
Here are the specifics of our analysis:
No robust ASLR: All devices.
No Secure Boot implemented: Devices 1 & 3 (Device 2 subject to further testing).
Unsigned firmware: Device 3.
No firmware encryption: Devices 2 and 3.
Overall, each device we tested is lacking in hardware security features that are considered essential for any modern computing device. In some cases, the devices had security enablement features that are insufficiently leveraged. Device 1, for instance, has Secure Boot support at the main processor level, but it does not actually implement Secure Boot, which ensures that a device boots only firmware created by the device manufacturer.
Signed firmware updates are another essential: This establishes the firmware’s integrity and that the device is installing the manufacturer’s original code, and that that code is genuine and has not been modified. Firmware encryption, which can protect against reverse engineering, provided the encryption key’s location is not easy for an attacker to find, is another feature we would expect to see on a mission-critical device. Yet only one of the three devices deployed signed firmware update files (results on one device are pending), and only one enabled encryption. Device 1’s update files are compressed with a ZIP password that effectively encrypts them, although it is not the most robust or secure encryption method.
Device 1 has cryptographic accelerator, a key to firmware checking, effective encryption and other functions, such as secure communications. Device 2’s integrated security engine effectively acts as a crypto accelerator since it offloads functionality from the CPU and supplies the computational power needed to enable cryptography.
Address Space Layout Randomization (ASLR) is another security measure that is becoming standard operation for many IoT devices, computers and phones. It protects against targeted attacks by continuously randomizing the location of firmware code in the device’s memory. Yet despite its value and prevalence of this protection, none of the devices we tested feature robust ASLR.
While most of the devices implement cybersecurity functions such as secure protocols, user access controls, port enable/disable and basics of firmware signing, none achieved a level of security appropriate for a mission-critical devices like protection relays.
Clearing a device’s debug tools and stripping its symbols deprives an attacker of tools that can help them determine how a device behaves or be used to spread an infection to similar devices in a network. Leaving these tools and symbols on the device can increase its overall functionality for the OEM’s support staff, but we categorize them as design flaws that decrease their resistance to attacks. Device 1 exhibited these flaws.
Similarly, tools compiled on a device can also provide an attacker with functionality to compile code or build their own programs on a device. Device 1’s Interactive VWorks Shell and C-Shell allows an attacker to interfere with OS tasks, memory extraction, script execution and other essential functions.
Undocumented ports, which Device 1 contains, are problematic on two levels. First, they can be a source of unmonitored activity on the device, effectively providing an attacker with an ingress that is not even part of the security apparatus.
If the ports are in fact monitored, non-malicious activity on them likely will register as an attack, leading to false-positive security alerts.
Our researchers were able to execute a remote code execution attack and two denial-of-service attacks on one of the devices. Fuzzing results on the other two devices will be updated pending ongoing efforts.
Some of devices have the potential for enhanced hardware-based cybersecurity, but none of the devices currently incorporates the full complement of security features. The good news is that the vendors certainly can make their device security more robust. However, given the limited regulation of embedded device security and the current state of the market, OEMs are not incentivized to improve the security profile of the devices they create, and so most are disinclined to commit resources to the effort, especially when their competitors are not. We therefore believe it is important to raise awareness of these security concerns and advocate for higher standards and more advanced security solutions.
Concurrent Red Balloon Security research confirmed that one of these protection relays is vulnerable to ransomware attack (See our article and proof of concept on this topic here). But even before this was confirmed, our analysis raised legitimate concerns about the security features of these devices. They fall well behind the security standards that govern PCs, phones, and other devices inside IT networks, and yet there is no good reason why they should not conform to these higher standards.
Given this state of affairs, additional security solutions will be relevant to all industries that depend on ICS and mission-critical embedded devices. Our research into other critical ICS and BMS devices demonstrate the urgent need for a systematic upgrade in cybersecurity and monitoring capabilities in the electrical grid, automotive, space equipment and other societal pillars.
Red Balloon Security’s monitoring and firmware hardening products are designed to address many of the challenges in the engineering of the protection relays we analyzed. We encourage you to learn more about our offerings here.
Ask any Texan who endured the power outages during a severe winter storm in February 2021 about the critical nature of the electrical grid, and you’re likely to receive an answer that is long on details of severe, even life-threatening hardship. We all depend on this critical energy source and infrastructure, yet operators are a long way from ensuring the protection of all components in electricity grids.
These conditions are not due entirely to a lack of regulations or industry standards. The Critical Infrastructure Protections of the North American Electric Reliability Corporation (NERC-CIP) are thorough and go a long way to ensure that OT systems in the electric grid are protected at the network and system level, and NERC has the capacity to hit violators with fines related to non-compliance.
However, NERC CIP regulations are not always applied to the distribution network, and do not currently provide sufficient guidance or high enough standards for securing embedded end devices, such as protection relays. While NERC CIP covers manufacturer’s patches and the enablement of security features, there are few mechanisms in place that would force a manufacturer to create a fundamentally more secure product. Many industry experts do recognize that embedded device security is below the maturity level it should be, given these devices’ importance to mission-critical systems, but this has not led to a concerted push for higher standards.
A number of factors contribute to this insecure stasis. Given the lack of regulation at this level and lack of demand from end users, device manufacturers are not incentivized to commit resources to developing more secure devices. While they may highlight specific security features or certifications their products earn, our research indicates that existing cyber certifications, while welcome, are not sufficient indicators of truly robust device security. The situation is rather like rating a car as “safe” without checking the condition of its tires, or some other essential component.
Regulations applied to different industries vary, but overall, there has been much more attention paid to the safety features of embedded devices rather than their security. While a “safety first” approach is reasonable, we argue that any discussion of embedded device safety that does not include a cybersecurity assessment is shortsighted at best, and a potential source of genuinely unsafe device operation.
To help bring awareness to the variability and gaps in safety-security standards for embedded devices, Red Balloon Security undertook a comprehensive analysis of three protection relays, each of which was released by a leading manufacturer and released after 2015.[1]
Each of these devices is currently available on international markets. Each highlights its cybersecurity features in its marketing specifications, including Safety and IEC63443 cybersecurity compliance, up to and including certification. And while this is far from an exhaustive analysis of the protection relay market, we believe our research is relevant to a general conversation about the electric grid devices (and also to embedded device security in many industries).
Protection relays play an essential role in the overall responsiveness and safe operations of electric grids. While far removed from the control room or IT systems, they are nonetheless accessible to determined attackers through several threat vectors (see our proof of concept for loading ransomware onto an embedded device).
The testing of these three devices included the following steps:
Here are the specifics of our analysis:
No robust ASLR: All devices.
No Secure Boot implemented: Devices 1 & 3 (Device 2 subject to further testing).
Unsigned firmware: Device 3.
No firmware encryption: Devices 2 and 3.
Overall, each device we tested is lacking in hardware security features that are considered essential for any modern computing device. In some cases, the devices had security enablement features that are insufficiently leveraged. Device 1, for instance, has Secure Boot support at the main processor level, but it does not actually implement Secure Boot, which ensures that a device boots only firmware created by the device manufacturer.
Signed firmware updates are another essential: This establishes the firmware’s integrity and that the device is installing the manufacturer’s original code, and that that code is genuine and has not been modified. Firmware encryption, which can protect against reverse engineering, provided the encryption key’s location is not easy for an attacker to find, is another feature we would expect to see on a mission-critical device. Yet only one of the three devices deployed signed firmware update files (results on one device are pending), and only one enabled encryption. Device 1’s update files are compressed with a ZIP password that effectively encrypts them, although it is not the most robust or secure encryption method.
Device 1 has cryptographic accelerator, a key to firmware checking, effective encryption and other functions, such as secure communications. Device 2’s integrated security engine effectively acts as a crypto accelerator since it offloads functionality from the CPU and supplies the computational power needed to enable cryptography.
Address Space Layout Randomization (ASLR) is another security measure that is becoming standard operation for many IoT devices, computers and phones. It protects against targeted attacks by continuously randomizing the location of firmware code in the device’s memory. Yet despite its value and prevalence of this protection, none of the devices we tested feature robust ASLR.
While most of the devices implement cybersecurity functions such as secure protocols, user access controls, port enable/disable and basics of firmware signing, none achieved a level of security appropriate for a mission-critical devices like protection relays.
Clearing a device’s debug tools and stripping its symbols deprives an attacker of tools that can help them determine how a device behaves or be used to spread an infection to similar devices in a network. Leaving these tools and symbols on the device can increase its overall functionality for the OEM’s support staff, but we categorize them as design flaws that decrease their resistance to attacks. Device 1 exhibited these flaws.
Similarly, tools compiled on a device can also provide an attacker with functionality to compile code or build their own programs on a device. Device 1’s Interactive VWorks Shell and C-Shell allows an attacker to interfere with OS tasks, memory extraction, script execution and other essential functions.
Undocumented ports, which Device 1 contains, are problematic on two levels. First, they can be a source of unmonitored activity on the device, effectively providing an attacker with an ingress that is not even part of the security apparatus.
If the ports are in fact monitored, non-malicious activity on them likely will register as an attack, leading to false-positive security alerts.
Our researchers were able to execute a remote code execution attack and two denial-of-service attacks on one of the devices. Fuzzing results on the other two devices will be updated pending ongoing efforts.
Some of devices have the potential for enhanced hardware-based cybersecurity, but none of the devices currently incorporates the full complement of security features. The good news is that the vendors certainly can make their device security more robust. However, given the limited regulation of embedded device security and the current state of the market, OEMs are not incentivized to improve the security profile of the devices they create, and so most are disinclined to commit resources to the effort, especially when their competitors are not. We therefore believe it is important to raise awareness of these security concerns and advocate for higher standards and more advanced security solutions.
Concurrent Red Balloon Security research confirmed that one of these protection relays is vulnerable to ransomware attack (See our article and proof of concept on this topic here). But even before this was confirmed, our analysis raised legitimate concerns about the security features of these devices. They fall well behind the security standards that govern PCs, phones, and other devices inside IT networks, and yet there is no good reason why they should not conform to these higher standards.
Given this state of affairs, additional security solutions will be relevant to all industries that depend on ICS and mission-critical embedded devices. Our research into other critical ICS and BMS devices demonstrate the urgent need for a systematic upgrade in cybersecurity and monitoring capabilities in the electrical grid, automotive, space equipment and other societal pillars.
Red Balloon Security’s monitoring and firmware hardening products are designed to address many of the challenges in the engineering of the protection relays we analyzed. We encourage you to learn more about our offerings here.
Ask any Texan who endured the power outages during a severe winter storm in February 2021 about the critical nature of the electrical grid, and you’re likely to receive an answer that is long on details of severe, even life-threatening hardship. We all depend on this critical energy source and infrastructure, yet operators are a long way from ensuring the protection of all components in electricity grids.
These conditions are not due entirely to a lack of regulations or industry standards. The Critical Infrastructure Protections of the North American Electric Reliability Corporation (NERC-CIP) are thorough and go a long way to ensure that OT systems in the electric grid are protected at the network and system level, and NERC has the capacity to hit violators with fines related to non-compliance.
However, NERC CIP regulations are not always applied to the distribution network, and do not currently provide sufficient guidance or high enough standards for securing embedded end devices, such as protection relays. While NERC CIP covers manufacturer’s patches and the enablement of security features, there are few mechanisms in place that would force a manufacturer to create a fundamentally more secure product. Many industry experts do recognize that embedded device security is below the maturity level it should be, given these devices’ importance to mission-critical systems, but this has not led to a concerted push for higher standards.
A number of factors contribute to this insecure stasis. Given the lack of regulation at this level and lack of demand from end users, device manufacturers are not incentivized to commit resources to developing more secure devices. While they may highlight specific security features or certifications their products earn, our research indicates that existing cyber certifications, while welcome, are not sufficient indicators of truly robust device security. The situation is rather like rating a car as “safe” without checking the condition of its tires, or some other essential component.
Regulations applied to different industries vary, but overall, there has been much more attention paid to the safety features of embedded devices rather than their security. While a “safety first” approach is reasonable, we argue that any discussion of embedded device safety that does not include a cybersecurity assessment is shortsighted at best, and a potential source of genuinely unsafe device operation.
To help bring awareness to the variability and gaps in safety-security standards for embedded devices, Red Balloon Security undertook a comprehensive analysis of three protection relays, each of which was released by a leading manufacturer and released after 2015.[1]
Each of these devices is currently available on international markets. Each highlights its cybersecurity features in its marketing specifications, including Safety and IEC63443 cybersecurity compliance, up to and including certification. And while this is far from an exhaustive analysis of the protection relay market, we believe our research is relevant to a general conversation about the electric grid devices (and also to embedded device security in many industries).
Protection relays play an essential role in the overall responsiveness and safe operations of electric grids. While far removed from the control room or IT systems, they are nonetheless accessible to determined attackers through several threat vectors (see our proof of concept for loading ransomware onto an embedded device).
The testing of these three devices included the following steps:
Here are the specifics of our analysis:
No robust ASLR: All devices.
No Secure Boot implemented: Devices 1 & 3 (Device 2 subject to further testing).
Unsigned firmware: Device 3.
No firmware encryption: Devices 2 and 3.
Overall, each device we tested is lacking in hardware security features that are considered essential for any modern computing device. In some cases, the devices had security enablement features that are insufficiently leveraged. Device 1, for instance, has Secure Boot support at the main processor level, but it does not actually implement Secure Boot, which ensures that a device boots only firmware created by the device manufacturer.
Signed firmware updates are another essential: This establishes the firmware’s integrity and that the device is installing the manufacturer’s original code, and that that code is genuine and has not been modified. Firmware encryption, which can protect against reverse engineering, provided the encryption key’s location is not easy for an attacker to find, is another feature we would expect to see on a mission-critical device. Yet only one of the three devices deployed signed firmware update files (results on one device are pending), and only one enabled encryption. Device 1’s update files are compressed with a ZIP password that effectively encrypts them, although it is not the most robust or secure encryption method.
Device 1 has cryptographic accelerator, a key to firmware checking, effective encryption and other functions, such as secure communications. Device 2’s integrated security engine effectively acts as a crypto accelerator since it offloads functionality from the CPU and supplies the computational power needed to enable cryptography.
Address Space Layout Randomization (ASLR) is another security measure that is becoming standard operation for many IoT devices, computers and phones. It protects against targeted attacks by continuously randomizing the location of firmware code in the device’s memory. Yet despite its value and prevalence of this protection, none of the devices we tested feature robust ASLR.
While most of the devices implement cybersecurity functions such as secure protocols, user access controls, port enable/disable and basics of firmware signing, none achieved a level of security appropriate for a mission-critical devices like protection relays.
Clearing a device’s debug tools and stripping its symbols deprives an attacker of tools that can help them determine how a device behaves or be used to spread an infection to similar devices in a network. Leaving these tools and symbols on the device can increase its overall functionality for the OEM’s support staff, but we categorize them as design flaws that decrease their resistance to attacks. Device 1 exhibited these flaws.
Similarly, tools compiled on a device can also provide an attacker with functionality to compile code or build their own programs on a device. Device 1’s Interactive VWorks Shell and C-Shell allows an attacker to interfere with OS tasks, memory extraction, script execution and other essential functions.
Undocumented ports, which Device 1 contains, are problematic on two levels. First, they can be a source of unmonitored activity on the device, effectively providing an attacker with an ingress that is not even part of the security apparatus.
If the ports are in fact monitored, non-malicious activity on them likely will register as an attack, leading to false-positive security alerts.
Our researchers were able to execute a remote code execution attack and two denial-of-service attacks on one of the devices. Fuzzing results on the other two devices will be updated pending ongoing efforts.
Some of devices have the potential for enhanced hardware-based cybersecurity, but none of the devices currently incorporates the full complement of security features. The good news is that the vendors certainly can make their device security more robust. However, given the limited regulation of embedded device security and the current state of the market, OEMs are not incentivized to improve the security profile of the devices they create, and so most are disinclined to commit resources to the effort, especially when their competitors are not. We therefore believe it is important to raise awareness of these security concerns and advocate for higher standards and more advanced security solutions.
Concurrent Red Balloon Security research confirmed that one of these protection relays is vulnerable to ransomware attack (See our article and proof of concept on this topic here). But even before this was confirmed, our analysis raised legitimate concerns about the security features of these devices. They fall well behind the security standards that govern PCs, phones, and other devices inside IT networks, and yet there is no good reason why they should not conform to these higher standards.
Given this state of affairs, additional security solutions will be relevant to all industries that depend on ICS and mission-critical embedded devices. Our research into other critical ICS and BMS devices demonstrate the urgent need for a systematic upgrade in cybersecurity and monitoring capabilities in the electrical grid, automotive, space equipment and other societal pillars.
Red Balloon Security’s monitoring and firmware hardening products are designed to address many of the challenges in the engineering of the protection relays we analyzed. We encourage you to learn more about our offerings here.
[1] Ethical vulnerability disclosure considerations prevent us from divulging the manufacturers of these devices.
[1] Ethical vulnerability disclosure considerations prevent us from divulging the manufacturers of these devices.
[1] Ethical vulnerability disclosure considerations prevent us from divulging the manufacturers of these devices.
Sal Stolfo was an original founding member of Red Balloon Security, Inc.
© 2022 Red Balloon Security.
All Rights Reserved.
© 2022 Red Balloon Security. All Rights Reserved.
Reach out to learn more about our embedded security offering and to schedule a demo.
Reach out to learn more about our embedded security offering and to schedule a demo.
Reach out to learn more about our embedded security offering and to schedule a demo.
Reach out to learn more about our embedded security offering and to schedule a demo.
