For many cybersecurity experts, Secure by Design – where protections are built in at the system design level – is the most future-proof approach to software security.This philosophy contrasts with the traditional use of external (overlaid) security measures applied to inherently insecure systems.
However, Secure by Design has not yet found widespread application and is mostly used in highly specialized systems, typically where physical safety is critical. Elsewhere in the industry, Secure by Design is considered an emerging approach that needs to be further development before widespread implementation.
The modern cyber world is undergoing rapid transformation, largely driven by artificial intelligence (AI). The very notion of security is expanding, making it increasingly difficult to cover all aspects. Traditional security measures must now account for new types of threats and vulnerabilities unique to AI-driven systems, such as adversarial attacks on neural networks or the manipulation of AI training data. This complexity makes it increasingly challenging to ensure comprehensive security, as each new AI capability can potentially introduce new risks.
In this dynamic environment, Secure by Design is no longer merely a desirable long-term aim. Rather, it is becoming an essential paradigm that must be applied today to keep pace with tomorrow’s changes.
In this post we will examine why integrating security at the design stage is critical to staying ahead of emerging threats. Specifically, we will explore how AI-driven innovations are reshaping the cybersecurity landscape and how the Secure by Design ideology can mitigate the cybersecurity risks posed by AI-based solutions.
Today’s technology landscape is characterized by the rapid convergence of all types of operational technology (OT) and information technology (IT). Homes have long been “smart” – and now cars, planes, transportation, industrial and control systems are following suit.
As a result, the nature of device networks is changing and becoming increasingly heterogeneous.
As IoT, and especially industrial IoT (IIoT), grows, system connectivity and availability are changing dramatically. Systems are becoming globally available, allowing individual devices and entire systems to be managed remotely from anywhere in the world.
The high connectivity of systems and their individual elements means that attacks and the effects of system downtime can spread more quickly (and sometimes unpredictably) than ever before. Additional IoT/IIoT threats arise from the challenge of updating device software itself, especially when such devices are numerous, geographically distributed, running heterogeneous software, or lacking a single source of updates. As a result, software updates are either irregular or nonexistent, or the costs associated with them are significant.
High-speed technologies such as LTE and 5G are shifting the focus from the transfer of content between users to high-speed communications between cyber-physical systems. In fact, this paradigm is directly embedded in the 5G architecture; its developers predict the number of connections will increase by three orders of magnitude over the next few years, with a similar increase in the amount of data transmitted.
Systems are becoming more complex due to the radical increase in the number of functions they perform. This, in turn, is leading to a significant rise in the amount of both system software code and reused third-party code. While code reuse is not new, it is now occurring on an unprecedented scale, shifting the focus from quantity to quality. The increase in the overall amount of software code, and especially reused code, makes it almost impossible to verify the correctness and security of a system.
The technological changes taking place around the world are not only exacerbating cybersecurity issues, they are reshaping and broadening the very concept of security. Today’s threats extend beyond information security to the realms of safety, privacy, reliability, resilience and more.
A breach in the IT part of a system can result in physical damage and failure of critical subsystems. This alters the nature of the risks involved, affecting not only data and the operation of information systems, but also the functioning of businesses and the lives, health and finances of individuals.
As a result, building cyber-physical systems without factoring in security at the design stage is increasingly risky, sometimes to an unacceptable degree.
In particular, the generally accepted “incident-investigation-countermeasure” approach is no longer appropriate in many areas because the damage from even a single (first) incident may be unpredictable and/or unacceptable.
Moreover, it’s not always possible to overlay additional protection on top of an inherently insecure system. For example, the design characteristics and underlying economics of IoT/IIoT devices often preclude the use of antivirus software, screens, monitoring tools, and more. The proliferation of AI plays a critical role in these processes, which will be discussed in the next section.
Artificial intelligence is commonly associated with neural networks. But neural networks are only one of a huge number of subtypes of AI. These other types of AI are built on different principles and architectures.
Any trivial program can be an “expert” system, performing “intelligent” actions that automate and/or replace human input. For example, deciding whether to apply the brakes of a car, open a hatch or raise the boom of a crane.
Even for more complex tasks, neural network AI is not always used. For example, artificial vision systems often use various nonlinear classifiers for pattern recognition, which calculate certain features and analyze them based on a large number of conditions. And clustering algorithms, such as the Gaussian Mixture Model (GMM), can be applied to identify anomalies in the operation of industrial equipment for early fault diagnosis. Also relevant here are fuzzy logic systems, simulated annealing algorithms, rule-based systems and algorithms based on colony behavior (e.g., the ant algorithm).
Decision-making in such (non-neural) systems is in fact based on multiple conditions. These can be very tricky to configure and test, but they are essentially verifiable – you can achieve 100% test coverage, build a formal model of a computational module, and prove its correctness.
Neural network AI is built on other mechanisms, and it is this subset that creates the most significant and fundamental problems with security in its modern, broad sense.
Unlike systems based on explicit conditions, neural network solvers cannot be 100% physically tested due to their combinatorial complexity. Their performance can only be checked statistically (probabilistically). Developers of neural network AI often admit they cannot fully explain their system’s decision-making because it is determined by millions of automatically adjusted coefficients.
Incidentally, the fundamental impossibility of 100% testing is also an inherent property of some non-neural systems. For example, systems that use genetic algorithms. Like neural networks, genetic algorithms are not verifiable because they use emergent properties of a complex system. In this sense, they are similar to neural networks. So, while it may be very difficult to prove the behavior of most non-neural (deterministic) systems is correct, it is possible in principle. However, for decision-making systems based on neural networks and for some non-neural systems (such as genetic algorithms), it is impossible due to the enormous combinatorial complexity and emergent properties of the system. This introduces risks that cannot be ignored.
In developing AI systems, especially those based on neural networks, there are peculiarities and risks that must be considered.
To create AI-based subsystems, it is critical to ensure the integrity and authenticity of all data used for training. The use of poor-quality or distorted data can lead to incorrect output from neural network models. And this can affect the security of systems that rely on that output.
It is impossible to guarantee 100% correct operation of decision-making units based on neural network AI. Their functioning is based on statistical models, and the very notion of “correct” neural network operation is complex and ambiguous. Furthermore, the amount of training and test data is often limited, making it difficult to verify the system’s behavior.
Distortion of input data during operation of the model (generation or recognition) can lead to unpredictable and erroneous output if these data fluctuations were not present in the training data sets. Neural networks can be sensitive to “unknown” anomalies in input data.
Attacks that manipulate neural network input data have already come to light, one of which is prompt injection. For example, neural networks in browsers have access to the browser API, which they can use to solve user tasks. In doing so, the neural network receives poorly controlled inputs, such as data from the page the user is viewing. Attacks have emerged in which a hidden (and artificially introduced) prompt on the page instructs neural networks to follow pre-prepared links, download files and perform other insecure actions on the user’s computer.
Neural network training and error correction are time-consuming and expensive processes. They require substantial computing resources and large volumes of labeled data.
Because of the high cost of training (see section 4), neural network AI is typically applied to complex, non-trivial tasks. As such, a wrong decision based on neural network AI can create major risks.
In today’s rapidly evolving technology landscape, implementing the Secure by Design philosophy is the most conceptually correct approach.
Note that there is nothing innovative about Secure by Design per se. It has been used for decades in critical cyber-physical systems where the level of security regulation is high. For example, in the aircraft industry, the integrated modular avionics (IMA) architecture concept is fully consistent with the principles of Security by Design.
Moreover, Secure by Design dovetails naturally with everyday life. For instance, electrical extension cords are designed from the outset to protect users from electric shock. Attempts to patch an inherently unsafe design at a later stage are usually ineffective (see the Secure by Design at Google report for real-world examples).
Another illustrative example is given by Kaspersky and borrowed from the construction industry. It can be called the “overhead hazard” principle. On a building site where cranes lift heavy materials, you can micromanage the boom, cable and load parameters as much as you like. Or you can establish a single rule for all workers: “Don’t stand under the boom!” In this case, everything else is secondary in terms of health and safety. The second approach is often more effective. We advocate a similar approach to software development, where the focus is not on analyzing every possible way the load could fall, so to speak, but on keeping key assets out of the danger zone in the first place.
In cyber-physical system development, the Secure by Design philosophy is not yet widely adopted. We see several reasons for this. First, there is the unfamiliarity of the Shift Left paradigm, which would allow the Secure by Design approach to be “toggled on”; as well as the habit of adding protection as an afterthought. Second, there is the perceived methodological immaturity of Secure by Design. And third, there is the additional cost that inevitably comes from integrating security properties into system architecture and design. In our experience, these additional investments at the initial stage can reduce the total cost of ownership (TCO) of the system, but this must be calculated on a project-by-project basis.
Ultimately, what is needed is a simple and cost-effective methodology that allows for the systematic implementation of Secure by Design for a wide range of cyber-physical systems in a range of industries. Kaspersky’s Cyber Immune approach to development claims to solve this problem.
The goal of the Cyber Immune approach is to create cyber-physical systems that possess Cyber Immunity, i.e., their declared assets are protected against undesirable events under any conditions, even under attack, subject to specified constraints.
The Cyber Immune approach consists of two parts aimed at providing methodological support for Security by Design:
This post does not include an implementation breakdown of the Cyber Immune approach. Instead, we will consider an important architectural feature of the Cyber Immune approach that helps combat the risks posed by AI.
As discussed above, systems with neural network AI cannot be 100% verified due to their combinatorial complexity. Therefore, by their very nature, they cannot be trusted components.
The remedy is to ensure that such subsystems are not able to place the system in an unacceptably dangerous state. This must be done at the architectural level.
That’s exactly what we do in the Cyber Immune approach. All components of the Cyber Immune system are divided into three groups, according to the set security objectives.
1. Trusted. These components have a direct influence on whether security objectives are met. Typically, these are components that directly operate or influence the system assets.
2. Highly trusted. These components increase confidence in the data flowing through them.
3. Untrusted. All other components.
The untrusted components are placed at the edge of the system, the trusted ones in the center, and the highly trusted ones in between.
In this arrangement, basic protection and verification are sufficient for untrusted components because the security objectives do not depend on them. The protection and verification requirements for trusted components are inherently high, but they can be mitigated because these components are “covered” by the highly trusted components and are somewhat in the background. It’s only the highly trusted components themselves, which are usually few in number, that need to be protected and verified with the utmost care.
AI must be considered an untrusted component. And specially designated trusted components check AI decisions from a security perspective to determine whether they could cause the system to transition to a dangerous state. If they do, the AI commands (decisions) are blocked.
So, we end up with a heterogeneous system in which only a small proportion of the components need to be carefully protected and checked, according to the Pareto principle.
This method is used to develop Cyber Immune solutions. It helps create a system that can be trusted, even though most of its components cannot be trusted individually.
In today’s rapidly evolving technology landscape, traditional methods of adding security measures to inherently insecure systems are no longer sufficient. We should embrace the Secure by Design philosophy to ensure that key assets are protected from the outset, reducing the risks associated with AI and other advanced technologies. And the Cyber Immune approach provides a comprehensive methodology for putting the Secure by Design ideology into practice. As we move forward, adopting a Secure by Design mindset will be critical to maintaining the integrity and reliability of our systems. By embracing these principles now, we will secure our technological future, allowing us, among other things, to fully realize the potential of artificial intelligence.
For many cybersecurity experts, Secure by Design – where protections are built in at the system design level – is the most future-proof approach to software security.This philosophy contrasts with the traditional use of external (overlaid) security measures applied to inherently insecure systems.
However, Secure by Design has not yet found widespread application and is mostly used in highly specialized systems, typically where physical safety is critical. Elsewhere in the industry, Secure by Design is considered an emerging approach that needs to be further development before widespread implementation.
The modern cyber world is undergoing rapid transformation, largely driven by artificial intelligence (AI). The very notion of security is expanding, making it increasingly difficult to cover all aspects. Traditional security measures must now account for new types of threats and vulnerabilities unique to AI-driven systems, such as adversarial attacks on neural networks or the manipulation of AI training data. This complexity makes it increasingly challenging to ensure comprehensive security, as each new AI capability can potentially introduce new risks.
In this dynamic environment, Secure by Design is no longer merely a desirable long-term aim. Rather, it is becoming an essential paradigm that must be applied today to keep pace with tomorrow’s changes.
In this post we will examine why integrating security at the design stage is critical to staying ahead of emerging threats. Specifically, we will explore how AI-driven innovations are reshaping the cybersecurity landscape and how the Secure by Design ideology can mitigate the cybersecurity risks posed by AI-based solutions.
Today’s technology landscape is characterized by the rapid convergence of all types of operational technology (OT) and information technology (IT). Homes have long been “smart” – and now cars, planes, transportation, industrial and control systems are following suit.
As a result, the nature of device networks is changing and becoming increasingly heterogeneous.
As IoT, and especially industrial IoT (IIoT), grows, system connectivity and availability are changing dramatically. Systems are becoming globally available, allowing individual devices and entire systems to be managed remotely from anywhere in the world.
The high connectivity of systems and their individual elements means that attacks and the effects of system downtime can spread more quickly (and sometimes unpredictably) than ever before. Additional IoT/IIoT threats arise from the challenge of updating device software itself, especially when such devices are numerous, geographically distributed, running heterogeneous software, or lacking a single source of updates. As a result, software updates are either irregular or nonexistent, or the costs associated with them are significant.
High-speed technologies such as LTE and 5G are shifting the focus from the transfer of content between users to high-speed communications between cyber-physical systems. In fact, this paradigm is directly embedded in the 5G architecture; its developers predict the number of connections will increase by three orders of magnitude over the next few years, with a similar increase in the amount of data transmitted.
Systems are becoming more complex due to the radical increase in the number of functions they perform. This, in turn, is leading to a significant rise in the amount of both system software code and reused third-party code. While code reuse is not new, it is now occurring on an unprecedented scale, shifting the focus from quantity to quality. The increase in the overall amount of software code, and especially reused code, makes it almost impossible to verify the correctness and security of a system.
The technological changes taking place around the world are not only exacerbating cybersecurity issues, they are reshaping and broadening the very concept of security. Today’s threats extend beyond information security to the realms of safety, privacy, reliability, resilience and more.
A breach in the IT part of a system can result in physical damage and failure of critical subsystems. This alters the nature of the risks involved, affecting not only data and the operation of information systems, but also the functioning of businesses and the lives, health and finances of individuals.
As a result, building cyber-physical systems without factoring in security at the design stage is increasingly risky, sometimes to an unacceptable degree.
In particular, the generally accepted “incident-investigation-countermeasure” approach is no longer appropriate in many areas because the damage from even a single (first) incident may be unpredictable and/or unacceptable.
Moreover, it’s not always possible to overlay additional protection on top of an inherently insecure system. For example, the design characteristics and underlying economics of IoT/IIoT devices often preclude the use of antivirus software, screens, monitoring tools, and more. The proliferation of AI plays a critical role in these processes, which will be discussed in the next section.
Artificial intelligence is commonly associated with neural networks. But neural networks are only one of a huge number of subtypes of AI. These other types of AI are built on different principles and architectures.
Any trivial program can be an “expert” system, performing “intelligent” actions that automate and/or replace human input. For example, deciding whether to apply the brakes of a car, open a hatch or raise the boom of a crane.
Even for more complex tasks, neural network AI is not always used. For example, artificial vision systems often use various nonlinear classifiers for pattern recognition, which calculate certain features and analyze them based on a large number of conditions. And clustering algorithms, such as the Gaussian Mixture Model (GMM), can be applied to identify anomalies in the operation of industrial equipment for early fault diagnosis. Also relevant here are fuzzy logic systems, simulated annealing algorithms, rule-based systems and algorithms based on colony behavior (e.g., the ant algorithm).
Decision-making in such (non-neural) systems is in fact based on multiple conditions. These can be very tricky to configure and test, but they are essentially verifiable – you can achieve 100% test coverage, build a formal model of a computational module, and prove its correctness.
Neural network AI is built on other mechanisms, and it is this subset that creates the most significant and fundamental problems with security in its modern, broad sense.
Unlike systems based on explicit conditions, neural network solvers cannot be 100% physically tested due to their combinatorial complexity. Their performance can only be checked statistically (probabilistically). Developers of neural network AI often admit they cannot fully explain their system’s decision-making because it is determined by millions of automatically adjusted coefficients.
Incidentally, the fundamental impossibility of 100% testing is also an inherent property of some non-neural systems. For example, systems that use genetic algorithms. Like neural networks, genetic algorithms are not verifiable because they use emergent properties of a complex system. In this sense, they are similar to neural networks. So, while it may be very difficult to prove the behavior of most non-neural (deterministic) systems is correct, it is possible in principle. However, for decision-making systems based on neural networks and for some non-neural systems (such as genetic algorithms), it is impossible due to the enormous combinatorial complexity and emergent properties of the system. This introduces risks that cannot be ignored.
In developing AI systems, especially those based on neural networks, there are peculiarities and risks that must be considered.
To create AI-based subsystems, it is critical to ensure the integrity and authenticity of all data used for training. The use of poor-quality or distorted data can lead to incorrect output from neural network models. And this can affect the security of systems that rely on that output.
It is impossible to guarantee 100% correct operation of decision-making units based on neural network AI. Their functioning is based on statistical models, and the very notion of “correct” neural network operation is complex and ambiguous. Furthermore, the amount of training and test data is often limited, making it difficult to verify the system’s behavior.
Distortion of input data during operation of the model (generation or recognition) can lead to unpredictable and erroneous output if these data fluctuations were not present in the training data sets. Neural networks can be sensitive to “unknown” anomalies in input data.
Attacks that manipulate neural network input data have already come to light, one of which is prompt injection. For example, neural networks in browsers have access to the browser API, which they can use to solve user tasks. In doing so, the neural network receives poorly controlled inputs, such as data from the page the user is viewing. Attacks have emerged in which a hidden (and artificially introduced) prompt on the page instructs neural networks to follow pre-prepared links, download files and perform other insecure actions on the user’s computer.
Neural network training and error correction are time-consuming and expensive processes. They require substantial computing resources and large volumes of labeled data.
Because of the high cost of training (see section 4), neural network AI is typically applied to complex, non-trivial tasks. As such, a wrong decision based on neural network AI can create major risks.
In today’s rapidly evolving technology landscape, implementing the Secure by Design philosophy is the most conceptually correct approach.
Note that there is nothing innovative about Secure by Design per se. It has been used for decades in critical cyber-physical systems where the level of security regulation is high. For example, in the aircraft industry, the integrated modular avionics (IMA) architecture concept is fully consistent with the principles of Security by Design.
Moreover, Secure by Design dovetails naturally with everyday life. For instance, electrical extension cords are designed from the outset to protect users from electric shock. Attempts to patch an inherently unsafe design at a later stage are usually ineffective (see the Secure by Design at Google report for real-world examples).
Another illustrative example is given by Kaspersky and borrowed from the construction industry. It can be called the “overhead hazard” principle. On a building site where cranes lift heavy materials, you can micromanage the boom, cable and load parameters as much as you like. Or you can establish a single rule for all workers: “Don’t stand under the boom!” In this case, everything else is secondary in terms of health and safety. The second approach is often more effective. We advocate a similar approach to software development, where the focus is not on analyzing every possible way the load could fall, so to speak, but on keeping key assets out of the danger zone in the first place.
In cyber-physical system development, the Secure by Design philosophy is not yet widely adopted. We see several reasons for this. First, there is the unfamiliarity of the Shift Left paradigm, which would allow the Secure by Design approach to be “toggled on”; as well as the habit of adding protection as an afterthought. Second, there is the perceived methodological immaturity of Secure by Design. And third, there is the additional cost that inevitably comes from integrating security properties into system architecture and design. In our experience, these additional investments at the initial stage can reduce the total cost of ownership (TCO) of the system, but this must be calculated on a project-by-project basis.
Ultimately, what is needed is a simple and cost-effective methodology that allows for the systematic implementation of Secure by Design for a wide range of cyber-physical systems in a range of industries. Kaspersky’s Cyber Immune approach to development claims to solve this problem.
The goal of the Cyber Immune approach is to create cyber-physical systems that possess Cyber Immunity, i.e., their declared assets are protected against undesirable events under any conditions, even under attack, subject to specified constraints.
The Cyber Immune approach consists of two parts aimed at providing methodological support for Security by Design:
This post does not include an implementation breakdown of the Cyber Immune approach. Instead, we will consider an important architectural feature of the Cyber Immune approach that helps combat the risks posed by AI.
As discussed above, systems with neural network AI cannot be 100% verified due to their combinatorial complexity. Therefore, by their very nature, they cannot be trusted components.
The remedy is to ensure that such subsystems are not able to place the system in an unacceptably dangerous state. This must be done at the architectural level.
That’s exactly what we do in the Cyber Immune approach. All components of the Cyber Immune system are divided into three groups, according to the set security objectives.
1. Trusted. These components have a direct influence on whether security objectives are met. Typically, these are components that directly operate or influence the system assets.
2. Highly trusted. These components increase confidence in the data flowing through them.
3. Untrusted. All other components.
The untrusted components are placed at the edge of the system, the trusted ones in the center, and the highly trusted ones in between.
In this arrangement, basic protection and verification are sufficient for untrusted components because the security objectives do not depend on them. The protection and verification requirements for trusted components are inherently high, but they can be mitigated because these components are “covered” by the highly trusted components and are somewhat in the background. It’s only the highly trusted components themselves, which are usually few in number, that need to be protected and verified with the utmost care.
AI must be considered an untrusted component. And specially designated trusted components check AI decisions from a security perspective to determine whether they could cause the system to transition to a dangerous state. If they do, the AI commands (decisions) are blocked.
So, we end up with a heterogeneous system in which only a small proportion of the components need to be carefully protected and checked, according to the Pareto principle.
This method is used to develop Cyber Immune solutions. It helps create a system that can be trusted, even though most of its components cannot be trusted individually.
In today’s rapidly evolving technology landscape, traditional methods of adding security measures to inherently insecure systems are no longer sufficient. We should embrace the Secure by Design philosophy to ensure that key assets are protected from the outset, reducing the risks associated with AI and other advanced technologies. And the Cyber Immune approach provides a comprehensive methodology for putting the Secure by Design ideology into practice. As we move forward, adopting a Secure by Design mindset will be critical to maintaining the integrity and reliability of our systems. By embracing these principles now, we will secure our technological future, allowing us, among other things, to fully realize the potential of artificial intelligence.
