Co-founder and CEO at 10 Senses
Since a while, we have been involved in AI Governance initiatives for Polish financial institutions. We are a member of a larger project team and at 10 Senses, our role in these projects has been primarily around technical aspects of AI Governance. This work focuses on what the AI Act describes under Article 17: the Quality Management System for high-risk AI systems.
In this article, we would like to share 4 lessons we have learned from real AI governance projects. These observations may be slightly different from what you usually find in general AI Act summaries or legal commentaries. They come from the practical side: from trying to make governance work inside organisations that already build, buy, and use AI systems.
Important disclaimer: we are not lawyers, and this article is not legal advice. Treat it as a technical and organisational perspective on AI governance. Any regulatory decisions you make should be made together with your legal, compliance, risk, and data governance teams.
1. AI Act specifies the outcome, not the process
One of the first practical lessons from AI governance projects is that the AI Act defines the expected outcome much more clearly than the path to get there. The regulation sets requirements for AI systems: what conditions should an AI system meet, how should it be governed, what level of oversight is required. In other words, it tells organizations what “good” should look like.
It does not, however, provide a step-by-step implementation manual. It does not tell you precisely what processes to design or which roles should be responsible for which decisions. That part is left to the organization. And this is where the real challenge begins. Many regulatory requirements are easy to agree with at the level of principle. It is hard to argue against risk management, human oversight or cybersecurity. The difficult part is turning these principles into a governance model that actually works inside a real company.
It is much easier to define the destination than to design the route. Coordinating people across various departments can be a pain. Writing rules that are specific enough to create control, but flexible enough not to block every AI initiative, is difficult. Creating documentation requirements that are useful rather than purely bureaucratic is not straightforward.
So the real question is: “How do we design AI governance that helps the organization meet these requirements in practice?”
This distinction matters. Poorly designed governance can become a burden. It can create confusion and generate documents no one reads. Ironically, when governance becomes too heavy or too unclear, it may also make compliance harder, because the organization loses visibility and control over what is actually happening.
Good AI governance should do the opposite. It should make the right path easier to follow. It should clarify responsibilities and help teams build, deploy, and monitor AI systems in a controlled way. The goal is to create a system that makes responsible AI development repeatable.
2. Best practices in AI development resemble those necessary for a high risk AI system
Another lesson is that many AI Act requirements are not as distant from modern AI engineering practice as they may initially seem. Over the last few years, a rapidly growing body of tools, techniques, and good practices has emerged around how to build and manage systems based on LLMs. This includes areas such as AI evaluations, guardrails, observability or traceability.
If we build an AI system that is meant to be used commercially, especially in a serious organizational context, we want these elements in place anyway. We want to know how the system performs and understand when it fails. We want to monitor usage, outputs, latency, cost, and quality. And this is where an important observation appears: once these practices are properly implemented, many of the expectations associated with high-risk AI systems become much easier to address.
So the conclusion is quite practical: if you develop AI systems according to strong engineering and governance practices, you will already be addressing many – perhaps even most – of the requirements that the AI Act places on high-risk AI systems. This does not mean that good engineering automatically equals legal compliance. It does mean, however, that the gap between “building AI properly” and “preparing for AI Act obligations” may be smaller than many organizations assume.
In practice, the best starting point is often not to create a parallel compliance universe. It is to strengthen the AI development lifecycle itself. Good AI Governance should not sit next to AI development. It should be embedded into it.
3. AI Act conflates AI models and AI systems
Another lesson is that the distinction between an AI model and an AI system matters much more today. The AI Act was developed in a period when AI meant by and large classical machine learning. In that world, the typical pattern was relatively straightforward: you train a model, validate it, deploy it, and then use it to generate predictions. In such a setup, the AI model and the AI system could almost feel like the same thing.
Today, with the rise of foundation models, and especially large language models, many AI systems are no longer built by training a model from scratch. The model is already available. It may be provided by an external vendor, deployed as an open-weight model, or embedded into a broader technical architecture. What the organization actually builds is not the AI model itself, but an AI system that uses the model in a specific context.
This distinction creates practical challenges for AI governance. A company needs to know what exactly is being governed under AI Governance rules. Is it the model, the application, or maybe is it the RAG pipeline? These questions are not theoretical. They affect everyday operations and compliance.
This is why a well-designed AI governance framework should explicitly define how the organization understands and governs AI models and AI systems. It should make clear where the model ends and where the system begins. It should also explain how governance applies when the organization does not train the model, but still builds or deploys a system that relies on it.
There is probably more than one reasonable approach to this issue. Different organizations may choose different setups. But the issue cannot be ignored. A precise and practical distinction between AI models and AI systems is one of the important building blocks of effective AI governance.
4. Data is only partially addressed in AI Act
The final lesson is that the AI Act only partially addresses the data-related challenges created by the rise of LLMs. The regulation does address data, of course. Article 10 introduces requirements related to training, validation, and testing data for high-risk AI systems. And article 12 requires record-keeping and logging capabilities.
These obligations make sense in the context of traditional machine learning. However, once we move into the world of LLMs, the picture becomes more complicated. If an organization uses an LLM, it usually does not train the model itself. The model is already available and is used as part of a broader system. This could suggest that data requirements related to model training are simply not applicable in the same way. But does that really mean that we can ignore article 10? It’s an open question.
AI systems based on LLMs can use data in much more flexible ways. In many real-life applications, the final output of a language model is strongly influenced by contextual data. Retrieval-augmented generation is a good example: the system retrieves relevant documents or database entries and passes them to the model as context. This has a decisive impact on the answer generated. So should such contextual data be treated as training data in standard machine learning? E.g. should it have its own quality, relevance, completeness, and bias requirements? The AI Act does not give answers to these questions.
There is also another challenge: output from LLM-based systems can vary significantly in format, length and structure. This creates additional governance questions around output quality, traceability, review, storage, and accountability.
The complexity increases even further when we consider the variety of data sources involved. A system may use internal documents, databases, user-provided inputs, search results, APIs, tools, and agentic workflows. In such systems, data is no longer just something used at the training stage. It becomes part of the system’s runtime behaviour. This reality is not described in the AI Act.
Whatever approach you choose, do not design these rules in isolation. Data-related AI governance should be developed together with the Data Governance team. Otherwise, there is a real risk of creating AI-specific rules that are inconsistent with existing data management practices.