TL;DR

• The process of developing, deploying and using AI applications consists of different steps. This includes choosing datasets and base models, implementing different training and learning methods and deploying the customized model. 
• In order to understand the potential risks that may be imposed by combining these actions in your business processes, you first need to understand the basics thoroughly. 
• The potential risks can vary from supply chain attacks or data leakage to runtime attacks such as prompt attacks, jailbreaks and resource exhaustion attacks.
• Visibility into your AI use cases is key to identifying potential pitfalls and mitigating risks. True risk mitigation can only be done based on a strong foundation of transparency and awareness to your own AI use cases and work methods.
• As the industry evolves, your security approach must evolve with it. New attack vectors and vulnerabilities are constantly emerging, as are the tools and methods for working with AI - this requires being adaptive and adjusting to any new directions and shifts, and can only be done with a comprehensive and holistic view over all your AI use cases.

‍

The AI Lifecycle - Mental Model

To start discussing the importance of AI visibility and the ways to perform it, we first need to have a quick dive into the process of developing and deploying AI models, and creating AI applications. This is a high-level scheme which describes the general AI life cycle - from the collection of basic building blocks required to get started to the final result of an up and running AI app:

Let’s break it down the scheme into the different AI assets it consists, dividing them into two types of processes - creating a custom AI app (marked in red), and creating an AI agent (marked in blue). Note that the description below refers to the main use cases of the AI life cycle, but of course doesn’t cover every edge case, so different processes may vary in certain aspects.

‍

Creating an AI application

1. Choosing the fundamentals
   
   • Base model choice - this decision should be taken based on your need for the model, and the different tasks it would need to fulfill. Different base models of different architectures may perform differently, and also respond better to different learning and training methods, therefore you should be minded to which model you choose to base your training process upon.
   • Training datasets choice and preprocessing - in order to customize your model to perform best for specific tasks, you are usually required to train it using customized datasets. Those datasets can contain internal or public data, and its purpose is usually to train the model based on a large number of examples of good and bad ways to make inferences.

2. Implementing the training process
   There are different ways to perform model training, either using different AI platforms (such as Amazon Sagemaker AI, Azure AI Foundry etc.) or implementing the training mechanism independently and running it locally. Using said platforms usually allows additional alternatives to using python notebooks for your training implementation, such as AI ML pipelines (no-code interface for creating the training logic) or using an interactive interface to run your code on the go. Those options are called ML workflows and are a main asset in the AI life cycle, as it has a major impact on the results of the training process and the final custom model.

3. Deploying your custom model
   After finalizing the training and evaluation process, the next step is to deploy the resulting custom model by creating a serving endpoint and connecting the model to it. The endpoint is the asset meant to allow the invocations of one or more AI models. The endpoint is responsible for networking and routing the requests, and the model itself is responsible for the inference - getting a request and generating the output. The endpoint will be called to by the finalized AI application, which can be done in different ways such as HTTP requests, API requests etc.

‍

Creating an AI Agent

The process of creating an AI agent is similar to the classic process described above, but is more simple in some aspects. Let’s break it down to the main steps:

1. Choosing the foundation model
   In this case, similar to the classic use case of AI apps, the architecture and structure of the model should match the purpose of the agent.

2. Applying tools and RAG mechanisms
   This is a step unique to AI agents, which includes attaching (and sometimes creating) different tools to your agents. Those can vary from generic abilities like web searches to specific computing skills you can give your agent. These tools are known to the AI model the agent is based on, and can be part of the inference process to allow fuller answers.
   A specific tool worth mentioning is connecting your agent to a RAG mechanism - which allows the agent to attach relevant context to every request to the model, based on internal information stored in a vector store.

The creation of the agent includes the serving endpoint, so there’s no need to actively create one and connect your agent to it.

‍

Why Do We Need Visibility?

Now that we understand the basics of the AI life cycle, we can point to security and compliance risks that may arise, both in pre-production and in runtime, and understand that visibility is key to identifying, mitigating and avoiding these risks.
Here are a few main examples of risks and mitigations via visibility and governance:

1. Supply chain attacks
   In most cases, companies who want to adopt AI into their business processes will use open source repositories to choose the base models and training datasets that best matches their needs. This exposes these businesses to various forms of supply chain attacks, like model files containing backdoors or other kinds of malicious code, and data poisoning which can lead to unwanted behavior by the model (either poor performance or even malicious unintended actions).
   Tools like model and dataset scanning can identify supply chain attempts and enable you to avoid using any malicious artifact in your environment.

2. Data leakage
   The use of internal and sometimes sensitive data as part of the training process or as part of runtime usage (RAG) can lead to data getting to unauthorized users, either in the case of employees being exposed to data they are not permitted to see or data leakage to users external to the company.
   Tools like runtime guardrails and authorization filters are required to assure no sensitive data is given to any unauthorized user.

3. DOS and resource exhaustion attacks
   In cases of unmanaged rate limits to your AI apps, agents and serving endpoints, you may be exposed to DOS and resource exhaustion attacks. This allows malicious users to make extensive requests to deny service to other users of the app, and make financial and operational damage by using a large amount of the company’s resources and quotas. 
   Visibility to any network and deployment configuration and detection of misconfigurations are crucial to preventing those attacks on your organization.

4. Prompt attacks
   This risk refers to any case of a malicious user succeeding to bypass the original instructions given to a model or an agent, and make it perform unintended actions. This category includes different methods such as jailbreaks and prompt injections. You can read more about those kinds of attacks in our last blogpost. 
   This risk can also be mitigated by runtime guardrails, ensuring the blocking of attempts to send malicious prompts to the model.

‍

The Future of Homegrown AI

As the world of AI continues to grow and becomes ever more relevant, the choice to adopt AI into business processes becomes more and more trivial. Having said that, it’s important to remember that this is still uncharted territory in many ways - platforms for AI development and deployment are still being shaped and changed at a very fast pace, new platforms and development methods are being released, new AI-based vulnerabilities and attack vectors are being discovered and the industry is always responding to them on the go.For these reasons, visibility and transparency is a critical part for any decision to be made regarding security of any AI use case. In order to keep up with the times, you must have full transparency and avoid shadow AI - keep your AI clear to keep your AI clean.

‍