Risk Management
Attack versus defence: how AI models can help both sides – Pt I
Everyone’s talking about AI, and with good reason. The burgeoning technology is already disrupting industries as diverse as healthcare, financial services and manufacturing. IT security vendors have arguably been among the earliest adopters of AI. But it’s not just the good guys that are investing heavily in new ways to utilise emerging models. Threat actors are also looking for opportunities to further their business or geopolitical goals.
Save to Folio
In this two-part series, we’ll take a look at developments on both sides, focusing specifically on two model types: categorising and generative AI.
How to use categorising AI effectively
Also known as “classifying”, “segmenting” or “descriptive” models, categorising AI does what is says on the tin: it takes an input and categorises it. An obvious application for network defenders would be in categorising inputs like network connections, behaviour or files into "good" and "bad". This is a logical continuation of the classic block and allow list—the only difference being that the “good/bad” decision is learned automatically rather than created manually.
However, caution should be exercised in a couple of areas. First, developers of such solutions should steer clear of overly binary decisions on whether an input is “good” or “bad”. There’s a big difference between a file judged to be 1% malicious versus one where the calculation is 49%. It might be useful to provide user-definable categorisations such as “potentially harmful” and “unwanted” to add more granularity around trust decisions.
Second, users need to know not only the answer but how an AI model arrived at it. Unfortunately, most current AI is not self-reflexive, i.e. it’s not aware of itself and its decision making and therefore cannot say why it has come to a particular decision. Today's categorising AI is therefore like a well-trained expert who evaluates the facts presented to them based on their gut feeling, in order to come to a conclusion. To mitigate potential risk here, it may be a good idea to store AI decisions for future forensics, and perhaps also the raw data on which those decisions were based. This in turn allows a human expert to reconstruct or revalidate the AI’s decision.
Driving generative AI success
The reason every boardroom is talking about AI today is because of generative models like ChatGPT, which have burst on the scene with an uncanny ability to interact with users in natural language. The model computes or "generates" an output from a huge pool of training data, combined with current context (questions or chat history). However, it’s important to remember that, despite their fluency in the user’s language, and their seeming ability to compose jokes, poems and other works of art, these models don’t really "understand" the content they learn. As such, everything that is produced is basically just a very good "remix" of the content the model was trained on—albeit a wealth of knowledge/content that no human could draw on in their lifetime.
Unlike categorising AI, the strength of generative models lies not with decision making but rather summarising and presenting information and facts in dialogue. If trained on the right data—for example, inputs covering network connections, interactions, and compliance and business requirements—they could emerge as extremely useful IT assistants. A generative AI model may be able to recommend optimised system settings, for example, or suggest priorities for a compliance strategy. With the right timely information it may even be able to deliver root cause analysis of attacks.
It’s important to remember, though, that the linguistic quality of its output is no indicator of the quality of the actual content. False positives are a common problem for some generative models, especially when they’ve only been trained on a small set of data. Some zero-day attacks may fly under the radar for this reason.
Trend Micro’s new generative AI-powered assistant, Companion, is different. It’s trained on strictly controlled, proprietary Trend Micro data and is designed for use by SecOps analysts overwhelmed by threat alerts and struggling to manage the workload with skills shortages. It will help users of all skill levels to be more productive by:
- Explaining and contextualizing alerts
- Triaging and recommending actions
- Decoding complex scripts
- Developing and testing sophisticated search queries
It’s already available in Trend Vision One, and we have plenty more planned.
Stay tuned for the second part in this series, where we’ll explore how threat actors are using AI to gain an advantage.