AI in Web Applications: The Most Common Integration Mistakes
An AI feature can impress in a demo and cause problems in everyday use. These integration errors lead to incorrect results, unintended access, and extra work.
A customer inquiry is automatically summarized. A document provides the data for a form. Employees ask questions about internal documents and receive relevant answers. These kinds of features can take work off people’s hands.
In an initial demo, this can often be shown quickly. In day-to-day use, incomplete information, conflicting documents, and users with different access rights come into play. That’s when you find out whether the feature has been reliably integrated into the application.
Integration covers the entire process, from input to use of the result. Connecting a model and displaying its response covers only part of that process.
1. The task is still too vague
An assistant is supposed to handle customer inquiries. This description leaves open whether it should summarize information, draft follow-up questions, or send a response itself. Each of these tasks requires different data and checks.
For an initial scope, I would describe the task more narrowly. For example: identify the issue, any order number mentioned, and any missing information in an inquiry. Your team reviews the result before it is used further.
Four points should be agreed on before implementation:
- Input: What text, files, or data does the feature receive?
- Output: What exactly should it return?
- Review: How can you tell whether the result is usable?
- Next step: Who uses it, and what happens to it?
This also helps you assess the value. If employees have to rewrite every summary from scratch, there is little time savings.
2. The model receives unsuitable information
An assistant is supposed to answer questions about your offering, but receives only a general description of the company. Current services, terms, and exceptions are missing. Even so, it can produce an answer that sounds convincing.
To answer questions using your own documents, relevant passages are often searched for first. These are provided to the model along with the question. This approach is called RAG—short for Retrieval-Augmented Generation.
Even then, the error can occur before the answer itself is generated. The search may find an outdated price list, miss an important footnote, or return documents about a different product.
That’s why both the documents and the search need to be checked. Which version is current? Do related pieces of information remain connected? Are relevant restrictions found? If there isn’t enough information to answer, the application should indicate that and allow a follow-up question or manual handling.
Displaying a source makes it easier to review the answer. On its own, it does not prove that every statement in the answer is supported by that source.
3. Answers are used without review
A model can return data in a defined format. For example, it can extract an order number, a date, and an amount from a document. This makes it easier to process results in the application.
However, a suitable format does not guarantee factual accuracy. The amount may belong to the wrong order or come from a different part of the document. The official documentation on structured outputs also points out this limitation. Source: Google
The application therefore needs its own checks. Does the order exist? Does it belong to the signed-in customer? Is the proposed change allowed at the current stage of processing?
AI feature output | Checks before use |
|---|---|
Data for a form | Format, required fields, and validation against the template |
Assignment to an order | Whether the order exists and whether the user is authorized |
Draft customer response | Content review and approval before sending |
Suggested status change | Permitted transition and required approvals |
For actions with significant consequences, I would start with suggestions that a person confirms. Actual results can then be used to assess whether further automation makes sense.
4. Access rights stop at the chat window
An employee is not allowed to open certain customer documents in the application. However, the assistant finds the same documents through a shared document search and repeats their contents in an answer.
This bypasses the existing access restriction. A prompt telling the model not to disclose confidential information is no substitute for a technical check.
Access permissions must be enforced when the information is retrieved. The model should receive only content the user is allowed to access for that particular request. Microsoft describes document-level checks for search and RAG applications. Source: Microsoft
Stored conversation histories and cached responses also need appropriate access restrictions. If access is revoked, that change must be reflected in connected data stores.
5. Documents can influence how the feature behaves
AI features often process content uploaded by users or obtained from other systems. That content may contain instructions that try to divert the model from its intended task. This is called Prompt Injection.
For example, an uploaded document might instruct the assistant to retrieve additional customer information and include it in the response. Such content must be treated as untrusted data. This is especially important when the model can use tools to read or modify data. Source: Anthropic
For the integration, I would translate this into clear boundaries: tools get only the permissions they need. Every access is checked by the application. Changes and external actions require an approval step depending on their impact.
An instruction to the model is not access control. The application must enforce which data can be read and which actions can be performed.
6. Testing uses only ideal examples
A well-written inquiry is great for a demo. In day-to-day use, information may be missing, customers may use different terms, or a document may contain several conflicting statements.
To evaluate the feature, you need a fixed set of test cases with defined expectations. These should include routine cases as well as cases where the feature should not provide a definitive answer. Anthropic recommends basing evaluations on specific success criteria and realistic tasks. Source: Anthropic
For an inquiry analysis feature, I would, for example, test:
- A complete inquiry with a clear order number.
- An inquiry without an order number.
- A message containing two different orders.
- A question whose answer is missing from the available documents.
- A file containing a manipulative instruction.
For each case, define what can be used and when a review is needed. Re-run these tests after changes to the model, instructions, or document search.
7. Delays and errors are not accounted for in the workflow
The user starts an analysis and sees no response for a long time. They click again. Now several requests are running in the background, and it is unclear which result belongs to which attempt.
The user experience is also part of the integration. The application should show whether a process is running, has finished, or has failed. Inputs must be preserved if an error occurs. For longer processing tasks, a background job can be useful, with a status that can be checked later.
Retries also need rules. If a message is sent or a record is created after an analysis, the same action must not be carried out multiple times.
Users need a working next step if the AI feature fails. This could be trying the analysis again, entering the information manually, or handing it off to an employee.
How I would prepare the integration
For the initial scope, I would choose a single task and describe its entire workflow. Then we define the required data, access rights, checks, and error cases. Only then can the feature be developed to fit into the existing workflow.
If you already have an idea, tell me what task the feature should handle and what happens to the result afterward. We’ll identify the data and integrations needed, and how to test the value before a wider rollout.