I spent the last ten days asking a reasoning model to translate 30 volumes of classical Chinese Zen records into literary Thai. The work surfaced a dozen useful ideas, and exactly one bad one. The bad one is the reason I’m writing this.

The good one is that translation quality, like security, is a property of a system, not a property of a prompt.

The text in question is the Jingde Chuandeng Lu (景德傳燈錄), compiled in 1004 CE by Daoyuan. Thirty volumes. About 1.4 million Chinese characters. Mostly dialogue between Chan masters, students, and the occasional emperor. The kind of text where a single misplaced Buddha-name turns a teaching into a doctrinal error.

My goal was not literary perfection. It was auditable, reproducible, segment-level faithfulness in a system that could be re-run, re-judged, and re-translated until the output was good enough to read aloud.

That second clause is the one most people skip.

The architecture problem nobody warns you about

The naïve approach is: paste Chinese into a chatbot, paste the Thai output into a Word document, repeat 800 times. This works for a poem. It does not work for a 30-volume canonical record, because the failure mode of a chatbot is silently confident divergence — it produces a fluent Thai sentence that says something subtly different from the source, and you have no way to find it without re-reading both.

The architecture I built looks like every other integrity-critical pipeline you’ve seen in security. Source text is normalized to JSONL, with one segment per line and a stable ID. Each translated segment gets a SHA-256 of the source, a SHA-256 of the target, and a unit_hash that binds them together. Segments roll up into a Merkle tree. Every release has a release_root_hash. The whole thing lives in git, with a CI workflow that fails the build if the recomputed root doesn’t match the committed root.

You don’t need a Merkle tree to translate a Zen record. You need a Merkle tree to re-translate one, because the second pass will disagree with the first on which segments are “done” and which are still “needs-review”, and you need a way to be honest about that disagreement at machine speed.

The other thing the architecture forces on you is prompt versioning as a first-class object. When the prompt changes, the segment hash changes, and the Merkle root changes, and you can see in git exactly which lines moved from draft to reviewed to needs-review as a result. When judge behavior drifts, the same is true. Without this, “we improved the prompt” is a story. With it, it’s a diff.

What “passing” actually means

Every pipeline needs a quality gate. Mine is a judge model that scores each segment on a 1–5 scale and flags anything below 3 as needs-review.

The first time I ran this end-to-end, the gate failed. 48% pass rate. The judge produced 100 detailed complaints about Buddhist terminology, Chinese characters leaking into the Thai, titles repeated three times in one paragraph, master lists truncated mid-name. The translation wasn’t broken — it was incoherent with the source text’s expectations. The model knew the words. It didn’t know the genre.

This is where the security-engineering reflex kicks in. When a vulnerability scanner flags 100 issues and 80 of them are real, you don’t throw the scanner out. You categorize the false positives, then you build a rule that suppresses only that category. The translation judge works the same way. The complaints fell into four clean buckets, and each bucket became one rule.

Rule one: Buddhist term accuracy is non-negotiable. When the model sees 文殊, it must produce Mañjuśrī, not a generic Sanskrit-sounding word. The failure mode I kept seeing was the model reaching for the most common Buddhist word it knew — Mañjuśrī got rendered as “Madhyamaka philosophy” in three different segs. The fix was a reference table: Chinese → Sanskrit/Pali → Thai. The model now looks it up, doesn’t guess. This is the same principle as a CVE name resolution table. Disambiguation is cheaper than confidence.

Rule two: titles translate once. The book’s title appeared three times in a single header. This sounds trivial until you remember that the Chuandeng Lu is a compilation of records, and every record starts with a header. If the header is bloated, the body never gets read. The fix was a hard rule: translate the title at the first mention, then refer to it as “this record” or “this scripture” for the rest. Simple regex check after translation: count title occurrences per segment, fix anything over one.

Rule three: zero Chinese characters in the Thai output. This is obvious in retrospect. I forgot to enforce it. The model would leave Hanzi mid-sentence because it didn’t know the Thai equivalent of “Yangzi” and figured leaving the Chinese was safer than transliterating. The fix was a one-line script that scans the output for CJK characters and either transliterates them with pinyin or rejects the segment for re-translation. Lesson: if your quality gate doesn’t check for it, it will leak.

Rule four: length completeness. The model would translate the first six masters in a catalog and stop, because the seventh master had a long name and the model ran out of conviction. The fix was a self-check: source character count × 1.5 ≤ Thai character count. For long catalogs, split the source in half, translate separately, concatenate. The asymmetry between source and target length in literary translation is real but bounded, and the bound is enforceable.

The loop that actually moved the needle

With those four rules embedded in the prompt, I re-translated the failing segments. Pass rate went from 48% to 64%. The four rules collectively fixed 47 of the 51 originally-failing segments. The remaining 4 are dense doctrinal passages that probably need a human.

The pattern matters more than the number. The loop is: translate → judge → categorize → codify → re-translate. Each cycle is bounded, each cycle produces a measurable improvement, and the rules are cumulative. v6 of the prompt contains the four rules above. v7 of the prompt added five more, derived from a stricter judge pass, including one for place names (潭州 transliterates as ถันโจว, not “Tan Prefecture”), one for dynastic year dates (大和八年 becomes “the 8th year of the Taihe era, 834 CE”), and one for anti-fabrication vocabulary (don’t invent a Thai word for 參禮拜 if the source says “to bow to”).

I rolled v7 back. Here’s why, and this is the bad idea I mentioned at the start.

The bad idea: more rules is better

I was so focused on the judge’s complaints that I wrote a v7 prompt with nine rules, a 70-entry glossary, and explicit examples for every failure category. The prompt was 15,000 characters. It contained, by my estimate, the entire operational knowledge of a graduate-level translator.

The model hung. Not crashed — hung. Four parallel workers, each with a 60-second timeout, would complete one segment, then sit at 0% CPU for two minutes. The reasoning model was so loaded with context that it couldn’t begin. I tried two workers instead of four. I tried sequential. I tried a smoke test of five segments. The 60s timeout failed 40% of the time. The model wasn’t slow because the prompt was bad. The model was slow because the prompt was too big for its context to reason over.

This is the lesson. There is a prompt size wall for reasoning models, and it is not advertised anywhere. I burned three hours to find it.

The fix was to throw v7 away. v6 with four rules gives 64% pass. v7 with nine rules would, in theory, give 75%. But if v7 takes five times longer to run and times out 40% of the time, the engineering cost exceeds the quality gain. The rational move is to ship v6, ship the PDFs, and cherry-pick the one or two most-impactful new rules for a v6.5.

I picked R7.1 (place names) and R7.5 (anti-fabrication). R7.5 in particular is cheap — about 200 characters of text — and it addresses the judge’s single most common complaint: “translation is completely unrelated to source”. I deferred R7.2-R7.4 because they add reasoning load, and I haven’t done the smoke test yet.

What I would tell someone starting from zero

Build the integrity layer first. JSONL with stable IDs, SHA-256s, a Merkle root, and a CI check that fails when the root doesn’t match. The cost is one afternoon. The payoff is that every other decision in the project becomes a diff in git, and you stop arguing with your past self.

Treat the judge as a separate model, not a function. Judge prompts drift. Judge outputs change. Judge scale calibration is its own research problem. A judge pass that says “all segs regressed 1 point” is more likely a judge-prompt change than a translation regression. Spot-check before re-translating.

Don’t optimize for the longest prompt that works. Optimize for the shortest prompt that ships. Four well-targeted rules in a 4,000-character prompt beat nine well-targeted rules in a 15,000-character prompt, because the second one breaks the worker pool and you never see the result.

The output of this project is 30 PDFs of Thai Zen dialogues, roughly 1.99 million Thai characters, generated over ten days of off-and-on work, with a v6 prompt that fits on a single screen. The system is the artifact. The translations will get better as the prompt iterates. The architecture is the part that doesn’t change.

If you’re going to use an LLM to translate a text that matters, design the system first, the prompt second, and the prompt-iteration loop third. The loop is where the actual work happens, and the loop only works if every previous iteration is auditable, comparable, and reproducible.

The rest is just a chatbot talking to itself.

What I did after I published this

I shipped the essay above and the v0.2.0-draft PDF together. Then I kept going, because two problems in the published artifact were bothering me.

The first was the judge score. The 2.73 average on Vol 4 was real but uninterpretable. The strict judge prompt was drifting from the looser one, and the 30-segment sample was too small to separate prompt noise from translation quality. The published PDF was a faithful rendering of the v6 output, but the quality claim I was making about it was softer than it looked.

The second was a single class of failure I had been deferring. v0.1.20 had flagged 119 segments as truncated — output cut off mid-sentence, missing the last few masters in a long catalog, that kind of thing. These were the worst-segregating segs in the corpus. I had treated them as “needs a human, ship around them.” That was wrong. They were a fixable problem, and shipping around them was leaving a third of the corpus in a known-bad state.

So I sat down and wrote a four-item Tier 1 fix list. The goal was a concrete, measurable lift: get the average above 3.0 and the pass rate above 15%, on a comparable judge prompt, over the segments I had flagged as broken.

Tier 1.1: re-translate the 119 truncated segments

The first instinct is to patch them. Don’t. The original v6 drafts were often plausibly written Thai that said something subtly different from the source. That’s the failure mode of a reasoning model with a truncation error — it commits to an ending it didn’t reach, and the prior reads as fluent.

The fix was to re-translate from source with max_tokens=8000 instead of the default 4000. The cost was 61 minutes of wall time across three parallel workers and one truncated 8147-character segment that needed a manual split. The result, re-judged with the v0.1.21 strict rubric: 119 segs, average 2.45, 48.7% pass at score ≥3, 7.6% at score ≥4.

The lift looks modest until you remember these are the worst segs in the corpus. Pre-fix, the v0.1.20 judge had put the pass rate at 0% on this set — every one of them was below 3. Post-fix, almost half are at 3 or better. The model wasn’t bad. The token budget was.

Tier 1.2: extract a master name glossary

The judge kept flagging master name errors. 大素禪師 was coming out as “ต้าซู่” (correct), “ดาซู” (acceptable), and “Da Su” (wrong) depending on the segment. The same master had three different romanizations in the same volume.

I extracted 218 unique master names from the existing translations and built a glossary mapping Chinese to Thai. The extraction was 90% right and 5-10% false positives — a few cases like 圓寂禪師, where 圓寂 is a verb (“passed away”) and the suffix 禪師 is the only real name token, came out wrong. The Thai dedupes handle this fine because the false positives are rare enough not to collide with the real entries.

The lesson: a glossary file is not a research project. It’s a script that runs in two minutes. The hard part is having a place to put it — a glossary hash field in the JSONL, a glossary reference in the prompt. Once that infrastructure exists, the glossary is a side effect of normal use.

Tier 1.3: re-translate the vols 13/14/15 catastrophe

I had been treating Vols 13, 14, and 15 as a localized problem. The judge scores were in the 1.8 range, which is bad even by this project’s standards. I assumed it was a “long master list” issue and that the corpus-wide fix would take care of it.

It wouldn’t have. When I dug into the JSONL, I found that all 72 of these segments had been marked status=draft from the very first batch and never re-translated. The v6 pass that re-translated vols 21/23/24 in the early loop had skipped them because they weren’t on the original failing list. The translations weren’t bad in an interesting way. They were bad in a boring way — written Thai that had nothing to do with the Chinese source. Classic “competent fabricator” failure, the model picking a plausible passage and rendering it well.

The fix was a clean re-translation with the v6.5-lean prompt on all 72. Re-judged: Vol 13 went from 1.81 to 2.58 average, with 19% to 58.3% pass at ≥3. Eleven of the 21 segments moved from score 1 to 2 or 3. The lesson is not “Vol 13 was hard.” The lesson is “the audit step found a regression I had not seen.” The audit step is the part you cannot skip.

Tier 1.4: the prompt size wall, revisited

The v7 prompt inflation I described in the previous section turned out to be even worse than I thought. I built a v6.5 prompt with all the v7 rules — 13,215 characters, everything I had learned, a Buddhist term table inline, anti-fabrication, place names, dynastic year formats. I ran a five-segment smoke test. One out of five completed. The other four hung at 0% CPU for 60 seconds and timed out.

There is a prompt size wall for reasoning models. It is not advertised. The wall for the model I was using sat between 9,800 and 13,200 characters of prompt. v6 fit. v6.5 did not. The fix was to throw away the inline Buddhist term table (14 entries) and refer to a glossary.yml file instead. The prompt shrank to 7,872 characters. The smoke test went from 1-of-5 to 5-of-5. The runtime went from “unusable” to “75 seconds per segment in parallel.”

I called it v6.5-lean. It is the production version. The lesson from the first essay is unchanged — more rules is not better — but the correction is: when the model can’t reason over your prompt, the answer is to externalize the reference data, not to delete the rules.

What the lift looks like, honestly

The honest read of the Tier 1 results:

  • 191 segments re-translated (119 truncated + 72 vols 13/14/15 drafts).
  • Vol 13 went from 1.81 to 2.58 average; pass rate from 19% to 58%.
  • The 119 truncated segs averaged 2.45 with 48.7% pass — better than zero, worse than I hoped.
  • The 2.45 is a partial judge number. It’s the v0.1.21 strict rubric on the T1.1 set only. A full re-judge of all 30 volumes is queued for v0.4.0.

The Vol 4 PDF was rebuilt from the updated JSONL. The new file is 721 KB instead of 788 KB. The 12-character hash prefix changed from 5a14316da0ab to 1cf1ed67df7b, which means the version registry gets a new entry, the old download link 404s, and the new one works. The file is T2076-vol04-1cf1ed67df7b.pdf. If you downloaded v0.2.0-draft, the link in your notes is broken. That is the point. I will keep saying this.

What didn’t work

The glossary extraction is full of false positives. About 5-10% of the 218 entries are wrong — usually from a suffix-stripping rule that ate too much (全禪師 → “ฉวน” is wrong; 全 is a name, not a generic prefix). A human pass is needed before the glossary is shippable.

The smoke test approach is too coarse. 5-of-5 at 120s timeout tells you the prompt doesn’t kill the model. It doesn’t tell you the prompt is good. A 50-segment calibration with per-segment timing distribution and per-vol coverage would be much better. I haven’t built that yet.

The split-retry pattern for the one catalog page that ran out of tokens is still a manual step. The model can translate a 1800-character master list if you split it in half and translate each half separately, but the split point is hand-picked, and the segments are stitched back together with a regex. This is fine for ten catalog pages. It is not fine for the 80+ catalog pages still in the corpus. Automation is overdue.

What I would do next

v0.4.0 has four items in priority order:

  1. A calibrated judge rubric. Pin the v0.1.21 strict prompt’s scoring distribution against a known-good reference set (50 hand-scored segments) so the 0–5 scale is comparable across runs. The biggest source of false-positive “regressions” right now is judge drift, not translation regression.

  2. Pre-emptive split for long catalogs. For any source segment over 1500 characters, split it in half before the model sees it. Translate each half. Concatenate. This kills the 1-of-119 truncation case at the root instead of catching it after the fact.

  3. Glossary pass with a human reviewer. Take the 218 entries, run the false-positive detector, present the borderline cases to a human with Thai-Chinese dictionary access. Aim for a 300-entry glossary that is 100% correct instead of a 218-entry one that is 90% correct.

  4. Full re-judge of all 30 volumes with v6.5-lean. The number I most want to know and don’t have: what is the v0.1.21 strict judge average on the full 807-segment corpus after Tier 1? My guess is 3.1 to 3.4. I will not know until I run it.

One thing I want to add to the previous essay

If I were writing the first version of this post now, the line about the judge would read: “A judge pass that says ‘all segs regressed 1 point’ is more likely a judge-prompt change than a translation regression. Spot-check before re-translating. And if the spot-check shows the judge drifted, fix the judge first.”

The fix-the-judge-first rule is the missing corollary. I was so focused on improving the translation that I trusted the judge output as ground truth. The judge is a model too. It has its own prompt, its own drift, its own failure modes. A pipeline with a single judge is a pipeline with a single point of failure. The next iteration of this project will run two judges in parallel and flag any segment where they disagree by more than one point.

— N