AA26-251A: "the model got dumber" is now something you measure, not something you feel

Joint Cybersecurity Advisory AA26-251A, published on 8 September 2026 by the NSA, CISA and the FBI, recommends that model providers answer suspected distillation traffic with a weaker model, and not tell the user. If your product's AI feature runs on someone else's model, "it feels worse this week" is no longer a mood. There is a dated document describing exactly that mechanism.

What the document does not do is prove that any provider has downgraded anyone. We haven't seen that proof either. So in our view the useful response isn't outrage. It's instrumentation: a pinned set of golden traces in CI, and the model ID recorded on every response.

What the advisory actually says

Three passages matter. They are quoted from the advisory on cisa.gov:

"using less sophisticated 'downgraded' models to respond to distillation requests"

"varying changes to responses across requests to complicate response quality evaluations, such that the subtle changes avoid triggering obvious alerts"

"Avoid informing… users suspected of distillation campaigns of a switch to a downgraded model"

The same guidance says AI safety researchers and third-party evaluators should be told about model changes. Everyone else, if flagged, should not.

The second passage is the one to read twice. The recommended degradation is designed so that a quality evaluation does not catch it. A spot check that reads a few answers and decides they look fine is exactly the test it is meant to pass.

Why a normal agent fleet fits the profile

The advisory lists indicators of suspicious traffic. Put them next to a production agent deployment and they read line for line:

advisory indicator                          typical production agent fleet
------------------------------------------  ------------------------------------------
shared account from multiple IPs / UAs      one service key, many workers, many hosts
24/7 load, no human pauses or idle time     queues drain around the clock
anomalous subscription-to-API ratio         almost all traffic is API, little chat
new subscription straight at maximum        launch day is a step change, not a ramp

None of this says you will be flagged. It says the profile the advisory describes and the profile of an ordinary agent product overlap. If a flag ever happens, the guidance points toward not telling you.

The honest version of the claim

A widely upvoted forum post has already accused providers of secretly degrading their models. Its author says openly that they have no independently verified benchmark, and the post shows no causation.

So we are not saying "providers are throttling you." The narrower claim holds up better: a documented recommendation now exists to swap in a weaker model without notice, tuned so quality evaluation won't catch it. With that on record, a feature that quietly gets worse could have four causes: your code, your prompt, your context, or which model served the request. Without instrumentation you can't tell them apart.

What catches a swap: the trajectory, not the final answer

A check on the final string is a weak test here. An agent can call tools in the wrong order, invent a parameter mapping, or miss a schema change without noticing, and still land on text that reads fine. The practice described in the source we rely on is to evaluate the trajectory instead: the exact DAG of tool calls, with inputs and outputs, run against a golden set of 30–50 verified real cases in CI on every change to a prompt, a tool schema or the model.

This is a sketch of that gate. It shows the described practice, not a measurement of ours:

# .ci/golden-traces.yml
golden_set: evals/golden/          # 30-50 verified real cases, pinned
run_on:
  - prompt change
  - tool schema change
  - model change
compare:
  - tool_call_order                # exact DAG, not "similar"
  - tool_call_args                 # parameter mapping per call
  - tool_outputs_consumed          # did the agent read what it was given
  - final_answer                   # checked last, weighted least
fail_if:
  - any trajectory diverges from its golden trace

The second piece costs almost nothing. Log which model actually answered, on every response, and assert on it:

resp = client.responses.create(model=PINNED_MODEL, input=task)

log.info("llm_call", requested=PINNED_MODEL, served=resp.model,
         trace_id=trace.id)

if resp.model != PINNED_MODEL:
    alert("served model differs from requested", trace_id=trace.id)

A caveat: a served-model field only reports what the provider reports. It rules out one class of silent change and nothing more. The golden traces are what catch a change that keeps the label the same. Taken together, "the model changed under us" turns into a failed CI run with a date, a trace ID and a diff.

What this costs, and where it goes wrong

This advice has a price, and it should be stated upfront. A golden set wired to a live paid endpoint and run on every commit behaves like production load that nobody owns. The source we rely on is a single anecdote from a vendor of token metrics, so read it as a place to look, not a statistic: one team's eval suite ran correctly for two weeks, spent the whole time, and the only alarm was the invoice.

What to check today:

1. which CI jobs call a paid model endpoint
2. which key each of them uses
3. whether that key has its own spending limit, separate from production

In our view, also give the eval suite its own key rather than sharing the production one. That keeps the bill readable. It also keeps your CI traffic from adding to a shared-account pattern.

FAQ

Has any provider been shown to downgrade customers under AA26-251A? Not in anything we have seen. The advisory is a recommendation dated 8 September 2026. The forum claims of degradation come without verified benchmarks, as their author admits.

Is the advisory aimed at ordinary businesses? It is written about distillation campaigns. The traffic indicators it lists (shared account, round-the-clock load, API-heavy usage, day-one maximum) also describe a normal production agent fleet. That overlap is the reason to measure.

Why 30–50 golden cases and not a large benchmark? That is the size range in the practice we cite: small enough to run on every prompt, schema or model change, and made of verified real cases instead of synthetic ones.

Isn't checking the final answer enough? No. An agent can take a broken path through its tools and still produce acceptable text. The trajectory shows the break, and the final answer hides it.

Won't running evals on every commit get expensive? It can, and silently. Put the eval suite on its own key with its own spending limit, and know which CI jobs call a paid endpoint.

A related case

Holt, a language-learning product in our cases, gates quality in a similar way: golden sets judged blind by a model with no hand in the material, rather than by eyeballing output. The case is at iloblique.com/holt. The short version of this piece is here: {{anchor_url}}.

Cases and process at iloblique.com.