The demo always goes well. You type a question, the model answers, the dashboard populates, and somebody says the word production. That moment is where most AI application security problems start, because an app that behaves beautifully for a friendly user tells you nothing about how it behaves for a hostile one.
What makes AI apps different is not that the code is worse. It is that the attack surface grew sideways while everyone was watching the model. There is now a component that accepts free-form text, calls other services on a user’s behalf, and produces output that downstream code treats as trustworthy. Each of those properties breaks an assumption ordinary web security relies on.
So the work before launch is less about hardening the model and more about hardening everything the model touches. Five areas cover most of it: the API surface, authentication, input handling, data privacy, and a round of testing that assumes somebody wants the thing to fail. None of it takes long before launch. All of it takes long afterward.
Lock Down the API Surface Before Anything Else
Every AI feature sits on at least two APIs: the one you expose to users and the one you call to reach the model. Both get neglected, and for opposite reasons. The outbound call feels like plumbing, so the key ends up in a config file. The inbound endpoint feels internal, so it ships without a rate limit.
Fix the outbound side by keeping provider credentials server side, with no exception for internal prototypes. A key shipped to a browser is a key somebody else can spend, and inference bills scale fast. Fix the inbound side with per-user rate limits and a hard cap on request size, because an unbounded text field is both a cost problem and a denial-of-service problem. Log which user made which call, since you cannot investigate abuse you did not record.
While you are there, narrow what the model’s tooling can reach. If an agent needs read access to one table, give it read access to one table. Teams running several agents at once learn this the hard way, which is why platforms for managing multiple AI agents now treat permission scoping as a headline feature.
Treat Authentication and Authorization as Server-Side Work
Authentication is where AI apps look finished and are not. Login works, the session persists, the chat history loads. Then a user edits a conversation ID in a URL and reads somebody else’s thread, because the ownership check lived in the interface rather than on the server.
Apply one rule without exceptions: every request that touches private data re-checks, server side, that this caller is allowed to touch this record. That includes requests the model makes on a user’s behalf, the part people miss. An agent using its own broad service credentials will happily retrieve a document the requesting human should never see, and nothing about the response will look wrong.
Use a standard flow rather than inventing one. Short-lived tokens, a refresh path, and scopes that constrain something will beat a bespoke scheme every time, and Retool’s write-up on how to secure AI apps covers what that looks like when governance happens at the organization level.
Validate Model Output, Not Just User Input
Classic input validation still applies: length limits, type checks, allowlists where the set of valid values is known. AI apps add a second obligation: validate the model’s output before anything else consumes it.
Model output is untrusted input. If it becomes HTML, escape it. If it becomes a database query, parameterize it. If it becomes a shell command, reconsider the design. And if your prompt includes content fetched from a document or a web page, assume that content may carry instructions aimed at your model rather than your user, which is the failure mode the OWASP Top 10 for Large Language Model Applications puts at number one.
Prompt injection has no clean fix, so the realistic goal is containment: separate system instructions from user content, keep the model’s permissions narrow, and require a human approval step for any action that spends money or deletes something.
Decide What the Model Is Allowed to Remember
Data privacy decisions get made for you if you do not make them deliberately. Logs capture full prompts, prompts contain customer records, and retention ends up at whatever the logging stack ships with.
Pick your answers before launch. What gets logged, how long it lives, who can read it, and whether anything is retained by the model provider. Redact obvious identifiers before they reach a third-party API, and read the provider’s data use terms. Where personal data drives automated decisions, the ICO’s guidance on AI and data protection is specific about the obligations that follow, including transparency about the logic involved.
The NIST AI Risk Management Framework earns its place here for a different reason. It gives you vocabulary and structure for the conversation, which matters once the people asking about risk are executives rather than engineers.
Test Like Somebody Who Wants It to Break
Happy-path tests prove the feature works. They prove nothing about whether it is safe. Write the unhappy ones: a request with no token, a request carrying another user’s token, an oversized payload, and a prompt instructing the model to ignore its instructions.
Run a dependency scan and read the output instead of filing it. Confirm error responses do not leak stack traces, internal URLs, or system prompt contents. Then hand the feature to somebody who did not build it and ask them to break it, because the author’s mental model is what hid the bug in the first place.
None of this is exotic. Most of it is ordinary application security applied to a component that happens to generate text, plus a short list of new concerns around output handling and model permissions. The expensive version is doing it after launch, once the fix has to be retrofitted around live users and real data.
Decide in advance what evidence you need before you ship. A written checklist covering keys, authorization checks, output escaping, logging policy, and the unhappy-path tests takes an afternoon to assemble and outlives every version of the app. Run it every release, not once.
The pace of AI development is not going to slow down, and it should not have to. What has to keep up is the judgment about what is safe to put in front of real people, and no tooling will take that off your hands.
