Our AI forecast lost to 'nothing changes'. Here is the test we now run on every forecast
An AI forecast is untested until it beats "nothing changes". Ours lost: it had been reading a cache frozen in May. Every RE Intelligence forecast is now scored against no-change, stops on stale data, and shows the user its error band.
The short answer: check any AI forecast against the dumbest rival there is, "next month looks like this month." Use one consistent metric. We think a model that can't beat that is not forecasting. It is narrating. We ran this check on the AI Forecasts module in RE Intelligence, our analytics product for the Dubai real-estate market. It lost almost everywhere. Every forecast in the product is now scored against no-change, refuses to run on stale data, and shows the user a ±X pp error band.
What follows is what broke, how we found it, and the five checks we now run. You can run the same five on your own product.
What happened
The forecast page kept rendering numbers that looked plausible. Nothing crashed and nothing turned red. That was the problem.
- The module read a sales cache frozen at 2026-05-17. Its "last month" was a partial May: 7,460 transactions where the source table had 16,739. The model read that gap as a -64.8% volume collapse and reported it.
- From 2026-07-11, community and project forecasts were not written at all. Their context queries took about 12.6 seconds against an inherited 8-second timeout. The n8n node was set to never raise an error, so only market-level rows came out.
- Another cache mixed mortgage transactions into prices, which produced a fake -13%.
We found it on 2026-11-09, and only because the owner asked a plain question: does the forecast work at all? No alert fired, because none existed.
We then scored the April–July 2026 forecasts. The forecast's "base" and the "actual" it was graded on turned out to be different metrics. Once we put both on one metric, the model lost to a no-change forecast almost everywhere. At project level its 3-month bias was +8.4 percentage points.
This is a class of failure, not just ours
AWS published an engineering write-up on evaluating a multi-agent system it built for global sales. In one measurement, data, tool code and the matching prompt change were meant to ship together but shipped at different times. That mismatch window alone dropped the pass rate from 78% to 38%. The write-up also notes that drift builds up even with no code release, because external data pipelines refresh every week.
Finance teams describe the same pattern. In one second-hand account, an AI workflow for a 600-row commissions model worked in May. In June new account codes arrived, it broke, and the AI kept saying its output was right. Practitioners in that discussion settle on one habit: whenever the inputs change, rerun a small set of examples whose answers you already know.
Our read (opinion, not measurement): models rarely fail loudly in production. The world under them changes, and they go on sounding confident.
The five checks
These are the checks we now run on every forecast in RE Intelligence. Each one closes a gap we actually fell through.
1. Base and actual are the same metric
We picked one canonical price series and score everything on it: 3-month median, Sales only, Unit + Villa. If the forecast starts from one number and is graded against another, the accuracy figure tells you nothing.
metric = median_price(
window = 3 months,
type = Sales,
property = Unit + Villa
)
assert forecast.base.metric == actual.metric2. Log the newest row the model actually read
Log the date of the newest data the model consumed, not the date the job ran. A job can run every night on a cache that stopped updating in May.
log(
run_at = now(),
newest_row_at = max(context.rows.date),
rows_read = len(context.rows)
)The fix also wired a cache refresh into the end of every data import, and raised the database statement timeout for this job from the inherited 8 seconds to 300.
3. Race against no-change
Accuracy is scored as the change against a naive forecast, not as an absolute error. The prompt is also anchored to no-change, with a cap on how large a move the model may predict.
naive_error = abs(actual - base) # "nothing changes"
model_error = abs(actual - forecast)
skill = naive_error - model_error # > 0 means the model earned its place
report(skill, by = [market, community, project])If the skill score is negative, the model is doing worse than doing nothing. In our view that number belongs on the dashboard you show internally, next to the forecast.
4. Fail closed
The workflow now has a data guard. It stops when a fetch fails, when the context is empty, or when the data is older than 45 days. It also trims a partial last month before the model sees it, which is the exact trap that produced the fake -64.8%.
if fetch_failed or context.is_empty:
stop("no forecast")
if today - newest_row_at > 45 days:
stop("data too old")
if last_month.is_partial:
drop(last_month)We think no forecast is better than a confident wrong one. A blank tile gets a question. A plausible number gets no question at all.
5. Show the user the error band
Every forecast now carries a "±X pp" band that the user can see. The read is still decision-grade: here is the expected move, and here is how wrong we have been. The call stays with the person.
show(
forecast = "+2.1 pp", # illustrative values
band = "±X pp", # from measured error vs actuals
)Where it stands now
After the rerun, 45 community forecasts, 45 project forecasts and 3 market forecasts are active and scored this way.
What we have not finished: the guard stops the run, but it does not yet send an alert when it trips. Our monthly and quarterly report workflows sit in the same risk class, and they are next. We would rather say so than pretend the job is done.
What would change our mind about this approach: if a forecast that never beats no-change still led users to better decisions, measured on outcomes, the race against naive would be the wrong bar. We have not seen that.
FAQ
Why compare against "nothing changes" rather than another model? Because it costs nothing, and in our experience it is hard to beat over short horizons. We think a forecast that cannot clear that bar has no business adding more sophistication on top.
Our forecast looks reasonable. Isn't that enough? Ours looked reasonable for six months while it read a cache frozen in May. "Looks reasonable" is exactly how a silent failure shows up.
Why fail closed instead of showing the last good forecast? The last good forecast was computed from data that is now wrong, and nothing on the screen says so. An empty state is honest and prompts someone to check. A stale number prompts nothing.
Won't an error band make users trust the product less? In our experience, people trust a stated error more than an unstated certainty. The band also gives them what they need to make the decision themselves.
Does this apply outside real estate? We think so, for anything that predicts from live data: demand, churn, cash flow, pricing. The AWS and finance examples above come from other fields, and the mechanism looks the same.
The case behind this
RE Intelligence is our own product: market reads for Dubai real estate, built and run by us. The full case is at iloblique.com/re-intelligence, and the product is at reint.ae. The five checks are also laid out as a short document here.
If you have something worth building, we'd like to hear about it.
Sources
- AWS Machine Learning Blog, "Building trust in AI agents: a production-ready evaluation framework with Amazon Bedrock AgentCore": https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore
- RE Intelligence engineering log, November 2026: forecast scoring, data guard and cache fixes (, internal).
- Finance-team account of an AI commissions workflow breaking in its second month: a public practitioner discussion, October 2026, paraphrased.