The AI Act is now in force and is reshaping the rules for every medical facility in the EU. If a physician pastes a visit transcript into the free version of ChatGPT, the facility is already violating several layers of regulations at once. We explain exactly what the AI Act says about the use of artificial intelligence in healthcare, what risks hospitals and clinics are taking on, and why using only proven, purpose-built tools is the only sensible path.
maximum fine under the AI Act for violations relating to high-risk systems
maximum fine under GDPR for unlawful processing of medical data
of global annual turnover — alternative fine threshold under the AI Act
minimum log retention period for a high-risk AI system
What is the AI Act and why it applies to every medical facility
The AI Act (Regulation 2024/1689) is the world's first comprehensive law regulating artificial intelligence. It adopts a risk-based approach — the greater the impact of an AI system on human health, rights and safety, the stricter the obligations. Healthcare was placed squarely in the category of high-risk systems.
Importantly, the AI Act does not regulate only model developers. It also covers so-called deployers — every facility that uses an AI system in its operations. A hospital, clinic or private practice in which a physician uses AI for documentation or clinical decision support becomes a responsible party before supervisory authorities.
When AI becomes a "high-risk system"
The AI Act — in Annex III and in conjunction with the Medical Device Regulation (MDR) — clearly identifies AI systems used for:
- supporting diagnosis and triage,
- analysing patient health data,
- generating medical documentation,
- assessing clinical risk or prioritising care,
- monitoring vital signs,
— as high-risk systems. It makes no difference whether the physician "only copied a snippet of the conversation into ChatGPT." What matters is the purpose of processing and the medical context.
Obligations of a facility that uses AI in healthcare
A deployer of a high-risk AI system must fulfil a set of binding obligations under Article 26 of the AI Act. The most important are:
- Human oversight — a designated, qualified person responsible for monitoring the system's operation.
- Log retention for at least 6 months, in a form that enables auditing.
- Informing patients that AI was used in the diagnostic or therapeutic process.
- Fundamental Rights Impact Assessment (FRIA) for public bodies and certain private entities.
- Use only CE-marked systems entered in the EU database.
- Compliance with the instructions for use provided by the supplier and incident monitoring.
Building this compliance layer around the free version of ChatGPT is practically impossible — because a consumer-grade LLM supplies neither logs, nor compliance documentation, nor a data processing agreement, nor any guarantee of where data is processed.
The AI Act does not replace GDPR — it operates alongside it
Many facility managers ask: "If we have GDPR, what do we need the AI Act for?" The answer is straightforward — both regimes apply in parallel and their sanctions are cumulative. Entering a visit transcript into the consumer version of ChatGPT, Claude or Gemini simultaneously causes:
- a breach of medical confidentiality (professional secrecy obligations),
- unauthorised transfer of special-category data outside the EEA (Articles 9 and 44 of GDPR),
- absence of a data processing agreement with the processor (Article 28 of GDPR),
- breach of AI Act obligations relating to high-risk systems,
- in some configurations — use of patient data to train models.
Real financial penalties
or 3% of turnover — AI Act fine for violations relating to high-risk systems
or 4% of turnover — GDPR fine for unlawful processing of medical data
or 7% of turnover — maximum AI Act fine for prohibited practices
Civil penalties also apply (patient claims), as well as professional liability for physicians and the risk of losing public health insurance contracts in the event of serious incidents. For many facilities, a single fine close to the maximum would mean the end of the business.
Why "free ChatGPT for notes" is an extremely costly saving
The most common objection from managers is: "Why do we need a dedicated assistant if physicians can paste the transcript into ChatGPT and get a note for free?" From an AI Act and GDPR perspective, this saving is illusory for at least three reasons.
1. Non-compliance = exposure to fines of up to €15 million
Using a general-purpose LLM for medical purposes immediately brings the facility into the high-risk system regime — without satisfying any of the AI Act's requirements. This constitutes open exposure to fines, investigations and the loss of operating licences.
2. Leakage of medical data and breach of medical confidentiality
Patient data entered into a consumer-grade ChatGPT leaves the EEA (typically ending up in the United States), is stored for up to 5 years and may be used to train models. This is a textbook violation of Article 9 of GDPR and of professional medical secrecy obligations.
3. No logs, no oversight, no documentation = no auditability
The AI Act requires the facility to be able to demonstrate who used the AI system, when and how. ChatGPT accessed through a physician's personal account generates no such logs. In the event of an inspection, the facility has nothing to defend itself with.
„AI systems used in healthcare should be designed and deployed in a manner that ensures an appropriate level of transparency, human oversight and accountability to patients."
Consumer LLM vs purpose-built medical assistant
| AI Act / GDPR requirement | Free ChatGPT / Claude | Purpose-built medical assistant (e.g. Cora AI) |
|---|---|---|
| Data processing location | Typically USA, outside the EEA | EEA only, in certified data centres |
| Data processing agreement (DPA) | Absent or limited, does not cover medical data | Full DPA signed with the facility |
| Training the model on patient data | Yes by default (free/Plus) | Never — contractual guarantee |
| Logs and auditability | None on the facility's side | Full event logs, exportable for auditors |
| Human oversight | Undefined | Physician authorises every entry |
| AI Act compliance documentation | None | Provided together with the system |
| Status in the event of an inspection | Multiple regulatory violations | Compliance demonstrable by documentation |
Why only verified, proven sources should be used
Artificial intelligence in healthcare is not "just another office tool." It is a technology that touches the most sensitive data in the entire economy — data about human health. For that reason, the choice of tool should be a deliberate management decision based on verifiable criteria, not on a physician's familiarity with a free consumer application.
A verified source is a supplier that:
- is a European entity subject to the AI Act and GDPR,
- processes data exclusively within the EEA,
- signs a data processing agreement covering health data,
- does not train models on patient data,
- provides full system logs and compliance documentation,
- designs the interface so that the physician remains the decision-maker (human-in-the-loop),
- publishes a security policy and undergoes independent audits.
How Cora AI addresses AI Act requirements
Cora AI was designed from the ground up as a dedicated medical assistant for the European market. This translates into specific technical and legal decisions that deliver regulatory compliance for the facility:
- EEA-only data processing — no transfers outside the EU.
- No training of models on patient data — guaranteed contractually.
- Full data processing agreement (DPA) covering special-category data.
- Complete system logs satisfying the 6-month retention requirement under the AI Act.
- Human-in-the-loop — every entry requires physician authorisation.
- Ready-made compliance documentation to hand over to the DPO and supervisory authorities.
In practice, this means that a facility using Cora AI does not have to build compliance on its own — it comes bundled with the tool.
Checklist for the facility manager
Before your physicians begin using any AI tool to work with patient data, verify:
- Does the supplier sign a data processing agreement covering medical data?
- Where is the data physically processed — is it exclusively within the EEA?
- Is input data used to train models?
- How long are transcripts and recordings retained?
- Does the system log events, and are those logs available to an auditor?
- Is there ready-made AI Act compliance documentation available for handover?
- Does the interface require physician authorisation for every entry?
- Does the facility have an internal "AI Policy" and a designated AI supervisor?
If the answer to any of these questions is "I don't know" or "no," the tool should not be used in patient care.
Summary
The AI Act is not an abstract regulation from Brussels. It changes the practice of every clinic in which a physician reaches for AI to document patient visits. Using free, consumer-grade models means exposure to fines of up to €15 million under the AI Act, up to €20 million under GDPR, and professional liability.
Safe use of AI in healthcare is possible — provided the tool was designed with these requirements in mind. A verified, purpose-built medical assistant takes the compliance burden off the facility and lets the team focus on the patient rather than on legal risk.