LLM Audit Trail: What to Record

LLM Audit Trail: What to Record

An LLM audit trail should record enough to reconstruct, later, which model and configuration produced an output, from which inputs and context, through which tool calls and human decisions, and what the system did next. In practice that means capturing identifiers, timestamps, versions and outcomes for every interaction, and storing full prompt and response text only where you can justify it. This guide lists the fields to record, explains how to handle sensitive content, covers retention under EU rules, and shows how to make the trail tamper-evident.

What an LLM audit trail is for

An audit trail is a chronological record that lets a reviewer who was not present reconstruct what happened and which system or person was responsible. For an LLM application, that reviewer might be a security investigator, an internal auditor, a customer’s lawyer or a regulator. Their questions tend to be the same: which model produced this, what did it see, what did it do, and did a person check it?

That is different from observability. Observability data such as latency, errors and token counts exists to keep a system running, and teams often sample it or let it expire. An audit trail exists for accountability, so it needs a defined retention period, restricted access and a way to show it has not been altered. The same instrumentation can feed both, but the requirements differ. This distinction is an editorial one, not a formal standard.

The core record for each interaction

The fields below are a practical baseline. They are Certelo’s editorial recommendation, not a mandated schema, so scale them to the risk of your use case.

Identity and time

* An event ID, plus a conversation or session ID, so entries can be linked and gaps noticed.

* Start and end time in UTC, taken from a synchronised clock.

* The application name and release version.

Actor and permissions

* A pseudonymous user or service identifier, the tenant, and the role.

* The data scopes or permissions the request ran with.

Model and configuration

* The provider, the model you requested, and the model version actually returned. A model alias can point to a newer version later, so a name alone may not be enough to explain behaviour.

* Generation settings such as temperature and output limits.

* The identifier and version of the system prompt and prompt template.

* The version of any guardrail or filter configuration.

Inputs and context

* The user input, or a reference and hash for it (see the next section).

* The identifiers and versions of retrieved documents or records, rather than their full text.

* The tools the model was allowed to call.

Actions and outputs

* For every tool call: the tool name, arguments, a reference to the result, success or failure, and any approval step.

* The output, or a reference and hash for it, plus the reason generation stopped and the token usage.

* Guardrail verdicts, including blocked or modified outputs.

* Errors and retries, so a failed attempt is not silently replaced by a later one.

Human involvement and consequences

* Who reviewed the output, what they decided (accepted, edited or rejected), when, and a reference to the edited version.

* What the system did next, such as sending a message or updating a record.

If you use OpenTelemetry, its semantic conventions for generative AI already standardise attribute names for the requested model, token usage and the reason generation stopped, and, when enabled, for system instructions, input and output messages and tool calls. The project describes these conventions as under active development, so pin the version you use. Aligning your field names with them keeps records portable between tools.

Store the content, hash it, or point to it

The hardest design choice is what to do with the text of prompts and responses. These often contain personal data, contracts, source code or credentials.

OpenTelemetry’s own guidance shows the default most tooling follows: by default, no prompt content or tool arguments are captured because they can contain sensitive data, and only metadata such as model names, token counts and durations is recorded. Content capture is switched on deliberately (OpenTelemetry).

That leaves three patterns:

1- Full content in the audit record. Easiest to reconstruct from, but every reader of the log can now read every conversation.

2- A reference plus a SHA-256 hash. The text sits in a separate, access-controlled store. The audit record holds a pointer and the hash of the exact text. If the stored copy is later edited, its hash will no longer match. If the stored copy is deleted for privacy reasons, the audit record still shows that a specific item existed.

3- Metadata only. Lowest exposure, but you cannot show what was actually said.

Many teams will use pattern 2 for most traffic and pattern 1 only for high-risk workflows. Two cautions apply. First, a hash of a short or predictable input can sometimes be guessed by trying candidates, so treat hashes of personal data as personal data until your privacy team says otherwise. Second, never log secrets such as API keys or passwords; redact them before anything is written.

Where logged content includes personal data, GDPR applies. Two of its principles bear directly on log design: data must be limited to what is necessary for the purpose (data minimisation), and must be kept in identifiable form no longer than necessary (storage limitation). Decide the purpose of each field before you collect it.

Retention and the EU AI Act

Jurisdiction: European Union. Information as of 16 October 2026. This is legal information, not legal advice.

If your system is classified as high-risk under the EU AI Act, logging is a legal requirement. Article 12 requires high-risk AI systems to technically allow automatic recording of events over the lifetime of the system, at a level of traceability appropriate to the system’s intended purpose (Article 12, EU AI Act). Article 19 requires providers to keep those automatically generated logs, to the extent the logs are under their control, for a period appropriate to the intended purpose and at least six months, unless other Union or national law, in particular data protection law, provides otherwise (Article 19, EU AI Act).

Article 12(3) sets a specific minimum for systems under point 1(a) of Annex III: the period of each use, the reference database the input was checked against, the input data that produced a match, and the identity of the people who verified the results. Check Annex III to see whether your system falls under that point.

On timing, the Digital Omnibus on AI, Regulation (EU) 2026/1744, was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026. It moved the application date for Annex III high-risk systems to 2 December 2027 and for Annex I systems to 2 August 2028 (DLA Piper, update of 10 August 2026; AI Act Explorer, Article 12).

Whether a given LLM application is high-risk depends on its intended use and the Act’s classification rules, so classify it first. Even outside the Act, sector rules, employment law, contracts and litigation holds can require longer or shorter retention. Retention in an audit trail is a legal and privacy decision, not only an engineering one.

Make the trail tamper-evident

A log written to the same system the application runs on can be edited by anyone with enough access. A common defence is a hash chain: each entry includes the hash of the previous entry, so changing one entry breaks every entry after it. That detects tampering, but it has a known gap. Someone who can rewrite the whole chain can recompute it, so the latest hash has to be recorded somewhere they cannot reach .

Options for placing that hash out of reach include a remote collector, a write-once store, an RFC 3161 timestamp authority, or a public blockchain anchor. Each rests on different trust assumptions. A blockchain anchor does not depend on your own organisation, which can matter in a dispute where your neutrality is questioned.

Certelo covers one specific step in this process. The workflow for LLM logs looks like this:

1- Seal each period’s audit records, for example daily, into one archive.

2- Calculate the archive’s SHA-256 fingerprint. Certelo does this locally in your browser, so the archive is not uploaded to create the proof.

3- Certelo anchors the fingerprint to the Electra Protocol blockchain and issues a PDF certificate.

4- During a review, hash the archive again and compare it with the recorded fingerprint. Anyone can verify a certificate without an account.

These steps follow Certelo’s description of its process (Certelo.io). A match shows that the archive existed in exactly that form at or before the recorded blockchain time.

It does not show that the log was complete, that its contents were accurate when written, who wrote it, that your clocks were right, or that the model really produced the recorded outputs. It is not a qualified electronic time stamp under the EU eIDAS Regulation, which is a separate legal category issued by qualified trust service providers; if your obligations require one, take legal advice. Keep redundant copies of the archive as well, because a fingerprint cannot rebuild a lost file.

A hypothetical record

The following is illustrative, not a real case. Field names are invented, and the values are placeholders.

Imagine a customer later claims the assistant promised a refund. The record would let a reviewer see which model and prompt version were in use, which policy version was retrieved, what the draft said, how the agent edited it and what was sent. It would not, on its own, show what the customer was told outside the system.

Gaps that quietly weaken an audit trail

* No model or prompt version. “The chatbot said it” is not reproducible if prompts and models change weekly.

* Logging only successes. Retries, refusals, blocked outputs and tool failures often matter most.

* Unsynchronised clocks. Event order across services becomes unreliable.

* Logs that live only on the application host. Forward them off the system as they are written.

* Retention that matches the storage budget, not the obligation. Decide the period from the purpose and the law.

* No verification practice. Test the check before an incident, not during one.

* Over-collection. Full prompts for every request create privacy exposure with little added value.