Your language app said “correct.” It scheduled the wrong word.

A language app that marks a near-miss as correct does not spoil one lesson. It spoils the schedule. The review you get next week is built on “you know this,” and you don’t. We found this in Holt, in two languages learners already use, and wrote it down in our own report. Then we replaced the hand-kept list that caused it with a test that checks the whole bank.

What we found

Holt is a language app that measures first and teaches second. It keeps a live map of what you know. Every skill carries its own mastery, stability and next due date, and each session is built from that map. Math owns the schedule; no model decides what you see next.

That design has one soft spot. The schedule is only as honest as the grade that feeds it.

Our latest build wave closed with a section headed “found in the LIVE pairs — not a regression of this wave, learners see it today.” It listed three things:

• 9 accent minimal pairs across two shipped languages sat outside the grading lists. A learner who answered the near-miss word was scored correct, and the wrong item was scheduled for review.
• 6 headwords in one shipped language printed a grammatically impossible article.
• A class of senses where a second answer is equally right: 472 of 6,001 senses (7.9%) in one pair, 528 (7.3%) in another.

An accent minimal pair is two words that differ only by an accent mark and mean different things. To a grader that compares loosely, they look like the same answer.

Why “correct” is the expensive mistake

Our view: a wrong “wrong” is annoying, and a wrong “correct” is worse. You notice the first one and argue with it. You never notice the second. The app agrees with you, and the schedule moves on.

The chain, simplified:

asked:      word A
answered:   word B        (differs from A only by an accent)
grader:     pair not on the list → accepted → "correct"
schedule:   the wrong item goes into review
learner:    still confuses A and B, and nothing will surface it

Nothing crashes. No monitor fires. The session looks fine. That is how it survived in languages that were already live.

How a new language found it

We did not find this by looking for it. We found it by building the next language.

Adding a language pair to Holt is not a translation job. One pair took 54 commits over seven days, phase by phase. It ended here:

entries              6,664
senses               7,242
examples            14,484
glosses             28,968
distractor edges    27,431
blind golden set    120 lemmas · 96.7% pass · 0.0% substantive defect
tests green          3,633

A new build re-reads every assumption the earlier ones froze. The new language needed its accent pairs handled, so the question came up of how the live languages handled theirs. The answer was a list someone maintained by hand, and the list was incomplete.

Our read: building the next instance of something is the cheapest audit you will ever run on the ones already live. It is also the report most teams file as scope creep.

The fix: a test, not a list

The change is small and boring, which is the point. The new language’s accent pairs are now guarded by a structural test that scans the whole bank. Nobody has to remember to add a pair.

A simplified sketch of the idea — not our code:

# before: the grader knows the pairs someone wrote down
ACCENT_PAIRS = [ ...maintained by hand... ]

# after: the check walks the bank itself and fails the build
for each headword in the bank:
    find every other headword that differs only by accents
    assert the grader treats them as different answers

A list is a record of what someone noticed. A test over the whole bank is a record of what is there. When the bank grows, the list goes stale quietly. The test goes red.

Same shape, second place

This was not a one-off. A separate review read the codebase as 35 areas, each by a reviewer with fresh context, with nothing fixed during the pass. It returned 109 findings at an average score of 7.26.

Fixing by class instead of by finding turned up something larger. Banned skills reached learners’ screens every day. The builder that assembles a session walked the table of skill states and never intersected it with the active skill graph. A shared helper for “which skills are alive” did not exist in the codebase at all.

before:  session = due skills from the state table
after:   session = due skills from the state table
                   ∩ skills alive in the active graph

Same lesson as the accent pairs: a protection existed where someone had looked, and was missing everywhere else it belonged. The same approach closed a swallowed-database-error class in ten more places than the review had named, including the guard at the entry of every learner page.

One more detail. 8 of the 35 areas were recorded as “not covered,” not as clean. “We did not look” and “there is nothing there” are different claims. For a learner, that is the difference between an honest app and a merely comfortable one.

Cleaning up without moving what you see

Work like this touches the word bank, so here is what it did not touch.

Counting duplicates honestly needed two independent projections of each record to agree. Measured that way, contamination was about 18% and 14%, where the count by headword had understated it two to three times. Coverage came out at 46.5% and 59.8%, against a documented floor of 80%.

The merge collapsed 773 surrogate nodes (4,440 → 3,667). Exact core duplicates went from 129 to 4, cross-language duplicate pairs from 301 to 1. The vocabulary a learner sees stayed identical to the row:

learner-visible senses   es 6,001 · en 5,936 · ru 5,779 · de 5,244
senses that lost their native side   0
left in production       rollback tables + a 748-row merge map

What is still owed

The structural test guards the new language. The two languages that shipped first are now openly owed the same test. We would rather say that here than call the problem closed.

Holt is still in development, and this is part of why. The bar is a grade you can trust.

FAQ

Did learners actually get wrong grades? Yes. In two shipped languages, a learner who answered the near-miss word of an accent pair was scored correct, and the wrong item was scheduled for review. Our own report says learners see it today.

Why didn’t the tests catch it earlier? The grading relied on a hand-maintained list. A list cannot tell you what it is missing. A test that scans the whole bank can, which is what the new language now has.

What about answers where two words really are both right? That is a separate class, and not a small one: 472 of 6,001 senses (7.9%) in one pair and 528 (7.3%) in another have a second answer that is equally right. Our view: a fair grader has to hold both lines at once — refuse the near-miss, accept the genuine alternative.

Is it fixed in every language? Not yet. The new language is guarded by the structural test. The two earliest languages are owed the same test, and we have said so.

Did the cleanup change my vocabulary? No. The learner-visible inventory stayed identical to the row in every language counted, no sense lost its native side, and rollback tables stayed in production.

The short version of this story is here: {{anchor_url}}. The full Holt case — how it maps what you know before it teaches anything — is at iloblique.com/holt, and you can try Holt at holt.iloblique.com.