Imagine a world where most of the code on the planet is free of bugs, where we can detect and fix vulnerabilities automatically and almost instantaneously, where diseases can be diagnosed and cured at the onset of minor symptoms or even before symptoms appear. These are some of the great advantages that artificial intelligence (AI) could provide.
The opposite, however, is also true. Criminals can use AI to create malware and phishing campaigns that are more sophisticated and targeted than the old “Nigerian Prince” emails we have seen before. Once infected, AI can be used to bypass protection mechanisms and detect where sensitive data is located. AI can also accomplish data exfiltration by using dynamic encryption or steganography to evade data loss prevention systems. As AI technology becomes increasingly integrated into diverse sectors such as finance, healthcare, and national defense, ensuring the security of these systems is paramount to prevent misuse, attacks, and unintended consequences.
Why should a mainframe professional worry about AI security or AI in general? Companies and governments are now in an arms race to create AI functionality that will allow them to leap ahead of their rivals. Mainframes should and will be part of this revolution. Mainframes remain the systems of records for most of the global economy. The new z16 mainframes now boast AI-on-chip capabilities and also use AI technology in their operating systems, subsystems, and applications for functions such as fraud detection, predictive analytics, and transaction processing.
In this article, I explore six challenges mainframers may experience as AI continues to gain prominence at their organizations.
1. Adversarial Attacks on an AI Model
Adversarial attacks involve techniques that target the vulnerabilities of AI algorithms by manipulating input data to deceive the AI model. These attacks can cause machine learning models to misclassify images. For example, adding colored patches to an enemy aircraft in order to trick the image recognition system into misidentifying them. These techniques are also used to bypass email spam filters, firewalls, and other intrusion detection systems.
We can see how an image recognition AI system can be tricked by the person on the right.
To protect against adversarial attacks, organizations should follow traditional cybersecurity practices such as patching known vulnerabilities and strong password management. Another method to protect AI models is by including adversarial examples in the algorithm training process. This approach helps the model learn to recognize and mitigate the effects of adversarial attacks. Other promising methods include implementing feature squeezing and defensive distillation.
2. Training Data Attacks Through Data Poisoning
Data poisoning occurs when attackers inject malicious data into the training dataset of an AI model or delete a portion of the dataset corrupting the learning process and causing the AI system to make incorrect decisions. Only a very small amount of data needs to be affected to adversely influence the whole model. This type of attack can significantly undermine the reliability and accuracy of AI systems.
Combatting data poisoning requires data sanitization techniques to clean and preprocess data, removing or mitigating the effects of potentially malicious inputs. Continuous monitoring and auditing of AI models as well as verifying the validity of data sources, especially if they come from a third party, should also be employed.
3. AI Model Output Attacks Through Model Inversion
Model inversion attacks aim to extract sensitive information about the training data from the AI model’s output. For example, an attacker might infer private details about individuals by querying a machine learning model repeatedly, feeding the model some of the output as input.
Differential privacy is a technique used to ensure that the outputs of AI models do not reveal sensitive information about individual data points in the training set. This is achieved by adding controlled noise to the data or the outputs, preserving privacy while maintaining overall accuracy. Homomorphic encryption, which allows computations to be performed on encrypted data, can prevent attacks if both the input and outputs of the model are encrypted.
4. AI Model Theft
Model theft, or model extraction, involves training a surrogate model or replicating a proprietary AI model by observing its outputs given certain inputs. Attackers can use this information to duplicate the model without having access to the original training data or model parameters.
Attackers can also extract some of the training data to create the surrogate model which can learn by sending predictions to the original model. If an attacker can predict what decisions the original model will make, they will know how to get around it. Stealing the functionality of an AI model is also a valuable trade secret that nation-state actors pursue.
To protect AI models, we can use techniques such as encryption, secure enclaves, and federated learning. Secure enclaves create a trusted execution environment, protecting sensitive information from unauthorized access or tampering. Federated learning is a technique that enables multiple devices to collaboratively train an AI model without sharing their data with a central server. The sensitive data stays on the devices and is not accessible by the central server which only aggregates and sends back an updated global model.
5. Privacy and Confidentiality Concerns
Although GenAI technologies, such as large language models and image generation systems, are ushering in a new era of unprecedented creativity, utility, and productivity, these models pose privacy and confidentiality challenges. Users might be feeding sensitive information, source code, or intellectual property to public GenAI systems such as ChatGPT, Gemini, or others.
Instead of banning or limiting GenAI, corporations can train users on what is allowed and not allowed as the correct tactic in combating risks. Using technical controls that proactively inform security teams if sensitive information is copied onto a publicly hosted GenAI model can also help with data leakage.
GenAI is also being used by attackers to create deepfakes, misinformation, and fake news that are affecting businesses’ reputations, as well as having social and political impacts. Unfortunately, misinformation is a challenging issue, even if something is identified as fraudulent, trust in real media may be undermined by false claims that real media is a deepfake.
6. Regulatory and Ethical Considerations
Organizations creating AI systems have, for the most part, relied on self-regulation without much oversight. Regulations, standards, and the red tape that comes with them are seen by some organizations as impediments to productivity and innovation; however, they are important and necessary to ensure safety and efficacy. Regulations and standards provide guidelines for the development and deployment of AI systems and make corporations accountable to regulatory entities. Governments are increasingly focusing on the security and ethical implications of AI. Frameworks and guidelines, such as the EU's General Data Protection Regulation (GDPR) and the AI Ethics Guidelines by organizations like the IEEE, emphasize the importance of transparency, accountability, and security in AI systems. The White House has introduced the Blueprint for an AI Bill of Rights, which provides an important framework for how government, companies, and citizens can work together to ensure AI systems are safe and trustworthy. The EU AI Act is the first concrete legal framework for regulating AI.
The Future of AI Security
AI is changing every day, which makes creating effective security measures difficult. Continuous research and innovation are necessary to keep up with evolving threats and ensure robust security frameworks. Proactive threat identification and ethical considerations will be key to safeguarding these systems against the growing array of threats. AI is a powerful technology, and we must harness its power, but to do that, we must manage and mitigate its risks.
Cesar Ulloa, CISSP, CEH, is a systems engineer who has supported mainframe customers of various industries. He has expertise in Mainframe HW, z/OS, Linux on z, Middleware, and security. He enjoys helping customers design IT mission critical systems by leveraging current investments and modernizing applications and infrastructure. Cesar was named regional designated specialist (RDS) in security and helped multiple organizations with their security-related issues and projects.