Introducing a framework for detecting, reacting and repairing threats to Machine Learning systems
Machine learning (ML) is used in most of the technologies today, affecting almost every aspect of our lives. As Machine Learning grows in importance, so does the number of tactics and techniques to circumvent them effectively. In October 2020, the MITRE organization, with the contribution of 11 companies, incl. IBM, NVIDIA, Bosch and Microsoft has developed Adversarial ML Threat Matrix, an open platform for the IT security industry. Its purpose is to enable security analysts to detect, react and remediate threats to Machine Learning systems.
Over the past four years, Microsoft has seen a significant increase in the number of attacks against commercial ML systems. Market reports also highlight this issue: Gartner’s Top 10 Strategic Technology Trends for 2020, released in October 2019, predict that “by 2022, 30% of all AI cyberattacks will use test data poisoning or theft of AI models.”.
Nevertheless, a survey by Microsoft showed a clear cognitive dissonance, especially among security analysts. They believe that the risk to ML systems is a futuristic problem. And that’s the issue because cyber attacks against ML systems are currently on the rise.
In 2020, we saw the first CVE published for an ML component in a commercial system, and SEI/CERT issued the first Vulnerability Notice, highlighting how many of the current ML systems may be subject to misclassification attacks that violate confidentiality, integrity, and availability ML systems.
The note proudly states:
„Machine learning models trained using gradient descent can be arbitrarily forced into misclassifications that may affect the items being classified. The impact of misclassification varies greatly depending on the purpose of the ML model and the systems of which it is part. “
The academic community has been sounding the alarm since 2004 and routinely demonstrates that ML systems, if not carefully secured, can be at risk.
Adversarial ML Threat Matrix was created because the response to the growing attacks on ML systems is to have a structure that systematically records the tactics and techniques used to breach the security of machine learning systems. Tabulated tactics and techniques can be effectively used to strengthen an organization’s monitoring strategies for critical ML systems.
Some of the most important assumptions about Adversarial ML Threat Matrix
- This is the first attempt at comparing known enemy techniques against Machine Learning systems. It is a living document that will be routinely updated.
- Only known evil is mentioned in The Matrix. Adversarial ML is an active area of research where new classes are constantly discovered. The authors are open to input from the security community.
- At the moment, the framework does not suggest defense measures as no consensus has been reached in this area.
- This is not a risk prioritization structure – A-ML Threat Matrix only compares known techniques; it does not provide the means to prioritize risks.
The goal of Adversarial ML Threat Matrix is to locate attacks on ML systems within which security analysts can gain an insight into these new and upcoming threats. The matrix has a structure similar to the ATT&CK framework, due to its wide application in the security analyst community, thus security analysts do not have to learn a new or different structure to learn about threats to ML systems. As Adversarial ML Threat Matrix is shaped like ATT & CK Enterprise, it follows the terminology: for example, the column headings are called “Tactics” and the individual items are called “Techniques”.
However, there are two main differences
- ATT&CK Enterprise is generally designed for corporate network which consists of many sub components such as workstation, bastion hosts, database, network hardware, active directory, cloud component and so on. ATT&CK Enterprise tactics (initial access, persistence, etc.) is really a key info about first access to the corporate network; durability in the corporate network. In Adversarial ML Threat Matrix, we admit that ML systems are part of the corporate network, we wanted to emphasize the uniqueness of attacks.
The difference: in the ML threat matrix, ‘tactics’ should be understood as ‘ML subsystem recognition’, ‘ML subsystem persistence’, ‘ML subsystem avoidance’.
- When we analyzed the actual attacks on ML systems, we discovered that attackers may use a variety of strategies: rely solely on traditional cybersecurity techniques; Rely solely on opposing ML techniques; or Use a combination of traditional cybersecurity techniques and ML techniques.
The difference: in Adversarial ML Threat Matrix “Techniques” come in two versions:
- Techniques in orange are specific to ML systems.
- The techniques in white are applicable to both ML and non-ML systems and are directly sourced from Enterprise ATT&CK.
The ML threat matrix is not yet part of the ATT & CK matrix.
The counter-ML threat matrix is also significantly different, as attacks on ML systems are inherently different from traditional attacks on corporate networks. Adversarial ML TM is based on data from real attacks on ML systems that have been confirmed by MITER as effective. It has also been discovered that attackers use a combination of “traditional techniques” such as phishing and internal spreading to attack the ML system, along with strictly ML-specific techniques. The main goal of the framework is to draw attention to the threats of attacks on the Machine Learning infrastructure and help structure the techniques of these attacks.
To learn more about it, please visit
Failure models in machine learning – Microsoft:
Public Statement by MITRE:
Github repository for Adversarial ML Threat Matrix: https://github.com/mitre/advmlthreatmatrix