Dear Reader,
On 17 August four researchers published a study of the tools meant to police whether AI models comply with regulation (“What Do Compliance Detectors Read?”, Sadhu, Sengupta, Sankarapu and Seth, arXiv). They took four deployed guard models and one ordinary language model put to the same task, and tested whether the verdict depends on the rule it is supposed to enforce. They deleted the rule, permuted it, replaced it with its opposite. The detectors’ accuracy did not change. The authors call the failure rule blindness.
A guard model is the layer a company puts between its AI system and the model’s output, to block what breaks its policy. Now that models do not only answer but act — calling APIs, sending documents, changing records — the guard is often the only place a bad action is meant to be stopped before it is carried out. In the compliance file it serves as a control. The study finds that at this point the tool does not respond to the content of the policy you gave it.
The rule said delete after 90 days; the guard passed 400
The tools tested are classifiers that check each output of a chatbot or agent and pass or block it: Llama Guard 3 from Meta, Qwen3Guard from Alibaba and WildGuard, alongside an ordinary language model in the same role and three probes that read the model’s internal state. The researchers ran them across twenty regulatory domains — among them GDPR, HIPAA, rules on crypto and money-laundering, and the platform policies of Google, OpenAI, GitHub, Reddit and X.
In one GDPR test the rule was: delete personal data after 90 days. The detector had to judge an output describing retention for 400 days, and should have blocked it. The researchers then deleted the rule, reversed it, or swapped it for another, and the verdict came out statistically the same. The detector was responding to how the situation looked, not to what the rule said.
One guard was built specifically to read the policy. It quoted the governing clause in its reasoning 91–95% of the time, yet when that clause was swapped for one permitting the same behaviour, the verdict changed in only 7.1% of cases. Even when the violated rule was removed entirely, the verdict flipped to “safe” in just 44.5%. It displayed the rule on screen and did not apply it. As the authors put it, citing a rule and adjudicating against it are separable, and these tools do the first.
The scale is the same across the set. In none of the twenty domains did deleting or shuffling the rule cause a statistically significant drop. On a 100-case HIPAA domain a detector scored 0.983 with the correct rule and 0.955 with the rule removed entirely — it was reading “violation risk” in general, not the specific requirement. On an anti-money-laundering test it did slightly worse with the rule than without it; the rule added noise, not signal. On ordinary inputs the accuracy of all the tools was fine, because the scenario alone usually points to the right answer, which is why nobody caught this earlier.
The mechanism is simple. A detector like this is trained on thousands of examples labelled violation or not, and it picks up a general “smell of risk” — the features of a situation that usually accompany a breach. At judgement time it reaches for that smell, not for the rule you gave it. When the scenario looks dangerous it blocks; when it looks innocent it passes, whatever the specific rule says.
This is not a separate flaw in these tools; it is how every large language model works. A model does not execute a rule the way code does. It predicts the most likely response from patterns in the data it was trained on. The same mechanism that makes it invent a plausible-sounding citation, or fill a gap with an assumption, is the one that makes it judge how a situation looks rather than check what a specific rule says. A model is good at soft judgement — “does this roughly look like a violation?” — and weak at hard, binary rule-application — “does this clause forbid exactly this output?”. So a model is poorly suited to being a strict rule-enforcer. The study only measured that.
Enforcement started, and the market is already selling this
The Commission gained powers to fine general-purpose AI providers on 2 August, and the high-risk duties, including credit scoring and recruitment, arrive on 2 December 2027. Tools for it are already on sale: Credo AI, Holistic AI, Lakera, and the cloud services from AWS, Microsoft and Google (Qovery).
They are sold as “AI Act ready” or “compliance monitoring”, and it is a guard like this that a firm points to in its file when a supervisor asks how it watches the model. The whole burden of proof then rests on a layer you have not tested in-house.
A guard that reads the rule and a guard that does not return the same result. The difference only appears at an audit, when you have to show that the safeguard in your file was applying the rule, not just recognising patterns.
The problem is not one product
This is not one weak tool. Every type tested was rule-blind: the guard classifiers, the ordinary model acting as judge, and three probes reading the model’s internal state. The authors even proposed their own simpler method, and on standard data it beat Llama Guard 3 — but it too responds to a general violation signal, not to the content of a specific rule. Their conclusion: a detector that cannot tell a scenario apart from the rule that governs it offers only the appearance of rule-specific assurance, whatever rule is in force.
For a buyer that means one thing: the problem is not solved by picking a better vendor from the same shelf. What is left is testing the tool yourself.
Impact on the Polish market
For a firm under KNF supervision, accountability is assigned directly: DORA and the outsourcing rules require you to evidence control over suppliers’ tooling, and such a guard is one of those tools. You are accountable for it, not the vendor. If it does not read the rule, your file describes a safeguard that is not really there.
Take a deployment that is ordinary today: a bank chatbot answering product questions, with a guard meant to block replies that look like investment advice or a promise of returns. A rule-blind guard blocks what sounds risky and passes wording that breaks the rule but does not look dangerous. On the dashboard the two look identical.
A simple test
Take a few dozen real outputs from your system and run them through the tool three times: once with your live policy, once with the policy deleted, once with the policy reversed, rewritten to permit what it should forbid. If the verdicts are the same across the three runs, the tool is not reading the rule and is not the safeguard you take it for. It takes an hour and needs no access to the model’s internals. Put the same test to the vendor: have them show, on your own policies, that the verdict responds when the rule is deleted or reversed, and that it still does after a model update.
Briefing
Google enters the legal-AI market. On 26 August Google Cloud launched Gemini Enterprise for Legal — a set of agents for contract analysis, legal research and regulatory monitoring, with secure connectors into law-firm systems and its first law-firm partners (The New Stack). It landed a day after Thomson Reuters, the incumbent owner of this market, unveiled its own legal model; OpenAI and Anthropic have had their own legal tools for months. The large providers are dropping from general assistants into specific, legally supervised work where a mistake has a price.
Anthropic values its market at $30 trillion. Ahead of its IPO the company told investors its addressable market is around $30tn — roughly US annual GDP, and about 40% of the entire US stock market — counting all the work that AI models could do (Fortune, 26 August). The figure tops the $28.5tn SpaceX showed investors before its own offering. Anthropic’s current run rate is about $65bn and its 2028 target is $190–200bn, under 1% of the claimed market. Fortune called the number outlandish and added that this may be the point: the prospectus is due shortly, with a debut possible as early as September or early October.
McKinsey says AI is “on the road to ROI”, but the results do not bear it out. AI spend keeps rising while the reported profit impact stays flat, with investment running ahead of an effect the earnings still do not show (The Register, 25 August). McKinsey keeps the optimism, but the phrase “on the road to” concedes that companies have not reached the end of it. A gap between spend and result eventually lands on the finance director’s desk, and the question becomes what exactly the outlay returned.
In summary
On 17 August researchers published a study in which four deployed guard models and an ordinary language model returned the same verdict once the rule they enforce was removed or reversed. One of them cited the right clause and applied it anyway. The Commission has been able to fine general-purpose AI providers since 2 August, the high-risk duties arrive on 2 December 2027, and tools sold as a safeguard for them are already on the market.
A safeguard that does not change its verdict when you swap the rule only confirms the tool was running, not that the rule is being enforced. Before you put it in the file, run it three times: with the rule, without the rule, and with the rule reversed. If the verdict does not change, the tool is not reading your rule.
Stay balanced, Krzysztof
Krzysztof Goworek is founder of Quintant — AI advisory that gets enterprises from experiment to production value.