.jpg)
According to What is the economic cost of covid-19?, the cost of the Covid pandemic could amount to $10tn in forgone GDP over 2020-21, making it the most impactful biological virus (so far). The accolade of most impactful computer virus according to The Top 10 Worst Computer Viruses in History | HP® Tech Takes is MyDoom which cost the global economy $52bn (inflation adjusted) and even now accounts for 1% of all phishing emails. Can an AI virus wreak similar damage? Or is AI immune?
Evidence shows that AI is very much susceptible to attack vectors (viruses). Famously In 2016, Microsoft’s Racist Chatbot Revealed the Dangers of Online Conversation when Twitter turned Microsoft’s innocent research chatbot into a horrible persona (so fitted well in the twittersphere). Today, you can download a makeup look from CV Dazzle: Computer Vision Dazzle Camouflage to defeat (a specific) CCTV face detection algorithm. In fact, AI may be more susceptible to viruses than conventional software.
The attraction of ML models that learns from the data presented (Unsupervised learning) is clear: labelling data accurately is labour-intensive and is normally done once at training so cannot react to model drift (IBM Watson Studio - Model Drift). If a model can learn from the data it processes, then we don’t have the cost of labelling, and we can continuously retrain our model to the latest data. By definition, the data is interpreted as software as it programs the model. Data poisoning is our first attack vector for an AI virus, and is extremely effective as Microsoft found out. It’s effective (and terrifying) because you don’t have to understand how the ML model works to be able to subvert it, so there is a trivial barrier to execute such an attack.
If you do understand how the model works, then so much the better. If you understand which features contribute most to the facial recognition, then you can design makeup that obfuscates that feature. If you understand which features contribute most to the ML model, then you can tailor the presentation of your data to achieve the decision that you want. Presenting counterfactual data (or lying) is our second attack vector, albeit one with a much higher barrier to execute because the lies need to be targeted to the specific ML algorithm.
Explainability is AI anti-virus
As with software or physical viruses, the first step in combating the virus is to understand the attack vector. This in turn requires understanding how the ML model arrives at its prediction. This is one of the reasons why we’re focused on Engineering Explainability into Oscar Enterprise AI; not just to quantify how a decision is made, but also what happens in the decision space leading up to the decision boundary.
By understanding the contributing features, AI practitioners can:
demonstrate which features solve the specific problem under consideration (and only utilise those features);
communicate how the problem is solved to users and regulators as well as fellow practitioners;
account for how these features solve the problem, and govern their utilisation.
It is this accounting and governing which is the next step in an AI anti-virus. When we know where the model is most vulnerable (the grey area), we can be vigilant to the effect of training data in this grey area, and perhaps add additional decision validation for grey input data.
We can therefore see that there are approaches that can be developed into tools to protect us from malicious misuse of ML models. I find it comforting that they are the very same tools that we need to build Responsible AI. It turns out that responsible AI is safer AI.