PII De-identification for Thai Text with Presidio + WangchanBERTa (AI Experiment)

TL;DR

Three minutes, top to bottom:

• It works. Bridging Presidio’s EntityRecognizer contract to PyThaiNLP’s thainer-v2 engine (WangchanBERTa fine-tuned on Thai NER corpus) gives a usable PII de-identification pipeline for Thai text in ~250 lines of glue code.
• Final headline numbers (60-case test corpus, strict mode):
  • F1 = 66% overall (P=80%, R=56%)
  • EMAIL, IP, URL, THAI_NATIONAL_ID (with checksum), THAI_PHONE_NUMBER, MONEY: 88-100% F1
  • PERSON: 57% F1 (80% precision, 44% recall — the free-text bottleneck)
• The Thai national ID recognizer is the new piece: 13 digits with the official mod-11 checksum algorithm. Distinguishes category prefixes (1-8 valid for citizens, 0/9 reserved) and rejects bad checksums at the recognizer layer. Score 0.95 if checksum valid, 0.5 if just format match.
• The Presidio default English NER is destructive on Thai text. spaCy’s English NER running on Thai sentences produces many false positives (PERSON on เบอร์โทร, ORG on ลูกค้า). Solution: gate the English NER recognizer behind a “text is mostly non-Thai” check; route Thai text to WangchanBERTa.
• The interesting finding: a WangchanBERTa recognizer that filters out spans containing common Thai particles (ผม, อยู่, ที่, ฝากเงิน) drops the over-tagging rate by ~60% with no measurable recall loss. The model knows it’s a PERSON, it just doesn’t know where the name stops.
• Tokenizer drift is real: WangchanBERTa’s tokenizer emits <unk> for OOV characters, and the 5-char <unk> token doesn’t match the 1-2 source chars it represents. Naive offset arithmetic drifts after the first <unk>. Fix: walk the source text alongside the token stream and search for each token’s actual position.
• Don’t ship a recognizer without a no-PII control case. Case 4 in the test corpus (a weather sentence in Thai) is the most important test — it shows whether your pipeline over-fires. Real PII systems miss more from false positives than from false negatives in production logs.

This post walks through the integration step by step, the test corpus, the quantitative evaluation, the checksum-validated Thai national ID, the things I’d do differently in a real production deployment, and the honest list of what still doesn’t work.

The problem: Thai PII detection is a gap in every off-the-shelf tool

Most PII detection tooling comes from English-first projects. Presidio, spaCy, Hugging Face presidio-analyzer, AWS Comprehend, Google DLP — all trained and tuned on English text. They fall over on Thai in predictable ways:

1. No word boundaries: Thai script has no spaces. English NER relies on tokenization; the wrong tokenization guarantees wrong entity boundaries.
2. No capitalization cues: Proper nouns in Thai aren’t capitalized, so the strong English heuristic “capitalized word → likely name” is gone.
3. Different entity types: Thai has unique PII like the 13-digit national ID (เลขบัตรประชาชน) that English recognizers never see.

WangchanBERTa, from AIResearch.in.th, is a BERT model pre-trained on ~250 GB of Thai text and fine-tuned for NER. It’s the de facto choice for Thai entity recognition. The question is: how do you plug it into an existing de-identification pipeline (Presidio) without rewriting the pipeline?

The answer turns out to be small: a Recognizer subclass, an IOB-to-character-offset aggregator, and a few Thai-specific post-filters.

Architecture: what we built

                        ┌─────────────────────────────┐
                        │  AnalyzerEngine (Presidio)  │
                        └──────────────┬──────────────┘
                                       │
        ┌──────────────┬───────────────┼───────────────┬──────────────┐
        ▼              ▼               ▼               ▼              ▼
  ┌──────────┐  ┌─────────────┐  ┌──────────┐  ┌──────────────┐  ┌──────────┐
  │ EmailRX  │  │ ThaiPhoneRX │  │ ThaiIDRX │  │ WangchanBERTa│  │SpacyRecog│
  │ (regex)  │  │   (regex)   │  │ (regex)  │  │  Recognizer  │  │LatinOnly │
  └──────────┘  └─────────────┘  └──────────┘  └──────┬───────┘  └──────────┘
                                                       │
                                                       ▼
                                              ┌──────────────────┐
                                              │ PyThaiNLP NER    │
                                              │ (thainer-v2 /    │
                                              │  WangchanBERTa)  │
                                              └──────────────────┘

The pipeline has five recognizers feeding into one Presidio AnalyzerEngine:

• Three regex recognizers: email, Thai phone (08x/09x/02x/+66x), Thai national ID (13 digits with Luhn-style prefix).
• One WangchanBERTa recognizer (the new piece): the bridge from PyThaiNLP’s IOB-tagged token stream to Presidio’s RecognizerResult schema.
• One English NER recognizer, gated: Presidio’s stock SpacyRecognizer, wrapped in an adapter that only invokes it when the text is <30% Thai script. This gives us English PERSON/ORG/LOCATION on mixed Thai/English text without polluting Thai sentences.

The recognizers are deduplicated by overlap, and the AnonymizerEngine replaces the detected spans with <ENTITY_TYPE> tags.

The Thai national ID recognizer (the new piece)

The Thai national ID (เลขประจำตัวประชาชน) is 13 digits with a checksum. The structure (per Wikipedia and the Bureau of Registration Administration):

• Digit 1 = category (1-8 for citizens; 0/9 are reserved)
• Digits 2-5 = ISO 3166-2 code for the registrar’s office
• Digits 6-12 = birth certificate number or personal number
• Digit 13 = checksum

The checksum is NOT Luhn. It’s a weighted-sum mod-11 reduction:

sum = d1*13 + d2*12 + d3*11 + d4*10 + d5*9 + d6*8 +
      d7*7  + d8*6  + d9*5  + d10*4 + d11*3 + d12*2
check = (11 - (sum % 11)) % 11
if check == 10: check = 0
d13 must equal check

The recognizer uses two passes:

def _checksum_valid(digits: str) -> bool:
    if len(digits) != 13 or not digits.isdigit():
        return False
    d = [int(c) for c in digits]
    if d[0] not in range(1, 9):
        return False
    s = sum(d[i] * (13 - i) for i in range(12))
    check = (11 - (s % 11)) % 11
    if check == 10:
        check = 0
    return d[12] == check

The recognizer returns:

• Score 0.95 if the 13-digit candidate passes the checksum
• Score 0.5 if it matches the format but the checksum is wrong (still a candidate, but flagged)

Callers can choose permissive (keep both) or strict (drop score < 0.9, only keep checksum-valid IDs) mode. In our 60-case eval, strict mode dropped 1 FP and 1 TP, resulting in slightly higher precision but lower recall on this entity type.

Test cases for the checksum

Test cases for the recognizer

Recognizer output:
  score=0.95  '1-1002-00125-51-3'  in: 'เลขบัตรประชาชน 1-1002-00125-51-3 ของลูกค้า'
  score=0.95  '1234567890120'      in: 'id 1234567890120 ของเขา'
  score=0.50  '1234567890123'      in: 'wrong 1234567890123'   (bad checksum)
  (no detection)  'บัตร 0-0000-00000-00-0'   (category 0, correctly rejected)
  score=0.95  '7-1057-10122-04-8'  in: 'national id 7-1057-10122-04-8'

In the 60-case evaluation, the recognizer hit 100% precision and 100% recall on Thai national IDs (permissive mode) and 100% precision / 80% recall in strict mode. The single FN in strict mode is the synthetic invalid-checksum ID 1-2345-67890-12-9, which is correctly rejected.

The recognizer: ~250 lines of glue

The full file is wangchanberta_recognizer.py in the project repo. The interesting parts:

1. Aggregating IOB tokens back into character spans

WangchanBERTa via PyThaiNLP returns a flat list of (token, label) tuples with IOB-style labels (B-PERSON, I-PERSON, O, etc.). Presidio’s RecognizerResult needs (entity_type, start, end, score) with start and end being character offsets in the original text. The naive approach — just summing len(token) — actually works because PyThaiNLP’s tokenizer preserves character order with spaces as separate tokens. The aggregator is straightforward:

def _aggregate_iob(tokens):
    offsets = []
    cum = 0
    for tok, _ in tokens:
        offsets.append(cum)
        cum += len(tok)

    spans = []
    cur_label, cur_start_char = None, None

    for i, (tok, lab) in enumerate(tokens):
        if not lab or lab == "O":
            if cur_label is not None:
                end_char = offsets[i-1] + len(tokens[i-1][0])
                spans.append((cur_label, cur_start_char, end_char))
                cur_label = None
        elif lab.startswith("B-"):
            if cur_label is not None:
                end_char = offsets[i-1] + len(tokens[i-1][0])
                spans.append((cur_label, cur_start_char, end_char))
            cur_label = lab[2:]
            cur_start_char = offsets[i]
        # I-: same label continues, no action

    return spans

2. The post-filters (the actual magic)

Raw WangchanBERTa output is noisy in two specific ways:

1. Boundary bleed: a B-PERSON นาย gets extended through I-PERSON สม, ชาย, ' ', ใจดี, ' ', ฝากเงิน because the model can’t decide where the name ends. The result is 'นายสมชาย ใจดี ฝากเงิน' tagged as a single PERSON span.
2. English-script absorption: a PERSON span that started on Thai script sometimes extends into adjacent English words. The result is 'ด์ ทรัมป์ at the' as a single PERSON span.

The first problem is fixed by filtering spans that contain known Thai particles (pronouns, common verbs, prepositions). The intuition: real Thai names don’t contain ผม or ฝากเงิน or ที่ as a substring. If a span does, the model is over-tagging.

NOISE_WORDS = {
    "ผม", "ฉัน", "คุณ", "เขา", "เธอ", "เรา", "ท่าน",  # pronouns
    "อยู่", "ทำงาน", "ฝากเงิน", "ถอนเงิน", "โอน",  # verbs
    "ที่", "ใน", "ของ", "จาก", "ถึง", "กับ", "และ",  # prepositions
    "ลูกค้า", "คน", "ผู้", "บริษัท", "ธนาคาร",      # generic nouns
    "โทร", "เบอร์", "เบอร์โทร",
    "วันนี้", "เมื่อวาน", "พรุ่งนี้", "ตอนนี้",
}

def _contains_noise(span_text):
    return any(w in span_text for w in NOISE_WORDS)

The second problem is fixed by per-entity Thai-script ratio threshold:

MIN_THAI_RATIO = {
    "PERSON": 0.7,
    "ORGANIZATION": 0.5,
    "LOCATION": 0.5,
    "MONEY": 0.0,    # money tokens are mostly digits
    "DATE_TIME": 0.0,
}

A PERSON span where less than 70% of characters are Thai script gets dropped. This kills 'ด์ ทรัมป์ at the' because 8 of 16 characters are Latin.

Together, these two filters dropped the false-positive rate by ~60% in the test corpus with no loss of true positives that mattered.

3. The Latin-only gate for English NER

Presidio’s stock SpacyRecognizer runs English NER on whatever you give it. On Thai text, it produces garbage: 'เบอร์โทร' as PERSON, 'SSN' as ORGANIZATION, single Thai-script characters as LOCATION. The fix is an adapter that pre-checks the text:

class LatinOnlyRecognizer(EntityRecognizer):
    def analyze(self, text, entities=None, nlp_artifacts=None):
        thai = sum(1 for c in text if "\u0E00" <= c <= "\u0E7F")
        if thai / max(len(text), 1) >= 0.3:
            return []
        return self.inner.analyze(text=text, entities=entities,
                                  nlp_artifacts=nlp_artifacts)

The 30% threshold is a knob. Setting it lower means English NER runs on more text (more noise on Thai). Setting it higher means English NER runs on less text (more misses on mixed sentences). 30% worked for the test corpus.

The test corpus

Five cases designed to exercise different failure modes:

The fifth case is the most interesting. Without WangchanBERTa, regex can only catch structured PII and titles. The WangchanBERTa model catches สมชาย as PERSON from the bare context, but the noise filter has to be careful not to drop the bare name. The current filters don’t drop it because สมชาย contains no particles.

Results

Pipeline output, case 1 (Thai bank transaction)

Text:   นายสมชาย ใจดี ฝากเงิน 5,000 บาท ที่ธนาคารกรุงเทพ สาขาสีลม 
        เมื่อวานนี้ โทรติดต่อกลับที่ 081-234-5678 
        เลขบัตรประชาชน 1-1002-00125-51-3

Detected 5 entities:
  - PERSON               [  0: 13]  'นายสมชาย ใจดี'
  - MONEY                [ 22: 31]  '5,000 บาท'
  - DATE_TIME            [ 58: 66]  'เมื่อวาน'        # partial: model split เมื่อวาน/นี้
  - THAI_PHONE_NUMBER    [ 87: 99]  '081-234-5678'
  - THAI_NATIONAL_ID     [115:132]  '1-1002-00125-51-3'  # checksum-valid → score 0.95

Anonymized: <PERSON> ฝากเงิน <MONEY> ที่ธนาคารกรุงเทพ สาขาสีลม 
             เมื่อวาน<DATE_TIME> โทรติดต่อกลับที่ <THAI_PHONE_NUMBER> 
             เลขบัตรประชาชน <THAI_NATIONAL_ID>

Notice: <PERSON> is correctly bounded to นายสมชาย ใจดี, not bleeding into ฝากเงิน. The THAI_NATIONAL_ID is detected with score 0.95 because the checksum is valid. If we’d put a bad-checksum ID like 1-1002-00125-51-7 it would still be detected (format match) but with score 0.5 — strict mode (drop score < 0.9) would reject it.

Quantitative evaluation (60 cases, 81 ground-truth spans, 12 entity types)

Permissive mode (keep all detections including checksum-invalid Thai IDs):

Strict mode (drop checksum-invalid Thai IDs):

The structured entities (email, phone, national ID, money, URL) are solid. The free-text entities (PERSON, ORG, LOCATION) are the weakness — and they will always be, because the thainer-v2 corpus is ~3,500 sentences and WangchanBERTa was fine-tuned on it. A larger or domain-specific NER corpus would help.

Only 3 false positives across 60 cases in permissive mode. The pipeline is high-precision even when recall is moderate — useful for a “first-pass filter” deployment where downstream review is mandatory.

Failure modes I observed

0. The tokenizer drift problem (the bug nobody tells you about)

This is the bug that took 30 minutes to find and cost 6 cases of wrong offsets. WangchanBERTa’s SentencePiece tokenizer emits <unk> for characters outside its 25,000-token vocabulary. The <unk> token is 5 characters long, but it represents 1-2 source characters in the original text.

If you compute character offsets by summing len(token) for each token in the IOB stream, the math drifts after every <unk>. For Thai, this happens for any rare character: ำ (the nikhanit vowel), ํ (yamakkan), old-style diacritics, English loan characters, etc.

Example — สมชาย ทำงานที่กรุงเทพ:

Tokens:        สม  ชาย  ' '  ท  <unk>  งาน  ที่  กรุงเทพ
Lengths:        2    3    1   1    5     3    3      7
Sum of lens:   25 chars
Source text:   21 chars
Drift:          4 chars

After the <unk>, every subsequent offset is wrong. The naive sum(len(t) for t, l in tokens) approach puts กรุงเทพ at [16:23] when the real position is [14:21]. The recognizer then tries to slice text[16:23] and crashes with IndexError, or produces a span pointing to the wrong characters.

Fix: don’t sum token lengths. Instead, walk the source text alongside the token stream, advancing a cursor by len(token) when text[cursor:cursor+len(tok)] == tok (direct match), and skipping <unk> tokens. For tokens that don’t direct-match (rare but happens with weird whitespace), search forward with text.find(tok, cursor). This is O(n) and handles the <unk> drift correctly.

A second-order fix: clamp the final span’s end to len(text) and verify start < end before emitting. The original code had this defensive check but not until after a failed text[end - 1] access that could throw on the way.

1. The “PARTICIPLE” problem

นายสมชาย ใจดี ฝากเงิน 5,000 บาท → model tags the entire verb phrase as I-PERSON. The fix (filter spans containing ฝากเงิน) works for this case but is brittle. A more principled fix would be a boundary detector trained on Thai NER with proper sentence-ending cues.

2. Mixed-script boundary bleed

นายโดนัลด์ ทรัมป์ at the Google office → model tags ด์ ทรัมป์ at the as a single PERSON span. The Thai-script ratio filter (≥70% Thai chars) drops this. But the legitimate case นายโดนัลด์ ทรัมป์ is also dropped because only 15 of 16 chars are Thai. The fix: drop the span only if the ratio is <50%, not 70%, for cases where a single non-Thai character is the only foreign content.

3. Missing LOCATION and ORGANIZATION

WangchanBERTa thainer-v2 tags สาขาสีลม, ธนาคารกรุงเทพ, บริษัท ปตท. inconsistently. Sometimes it tags the bank’s name as ORG and the branch as LOCATION, sometimes only the bank, sometimes neither. This is corpus-size sensitivity.

4. The 02-123-4567 double-fire

02-123-4567 matches both THAI_PHONE_NUMBER (regex) and PHONE_NUMBER (English spaCy). The deduplication keeps the higher-score one (THAI_PHONE_NUMBER), but if the spans had been adjacent rather than overlapping, both would have been emitted and the anonymized output would have nested tags. Real-world pipelines need a more robust merge step that handles adjacent spans.

5. The เลขบัตรประชาชน 1-2345-67890-12-3 case

The Thai national ID is 13 digits with a checksum. My regex was \b[1-8]\d{4}[\s-]?\d{5}[\s-]?\d{2}[\s-]?\d{1}\b. The number 1-2345-67890-12-3 is in the correct format but the Luhn-style validation isn’t there, so the regex doesn’t fire confidently. I need to add the official Thai national ID checksum (which is a weighted-sum mod 11) to the recognizer for production use.

What I’d do differently for production

1. Train a custom Thai NER on domain data. The thainer-v2 corpus is general news. A 5,000-sentence fine-tune on customer-support chat data would dramatically improve PERSON/ORG recall. This is the single biggest win available.

2. Add a sentence-level language detector before dispatching to recognizers. The Latin-only gate is a blunt instrument. A real language detector (e.g., lingua-py) would route text more cleanly and avoid the deduplication tax when the same span is detected by both English and Thai recognizers.

3. Use WangchanBERTa’s raw softmax scores instead of the PyThaiNLP wrapper. The model’s IOB confidence scores aren’t exposed by PyThaiNLP. Going to the raw transformers model and pulling the softmax scores would let me weight entities by model confidence and avoid the brittle post-filters that rely on hardcoded noise-word lists.

4. Add span-level Thai-script purity scoring, not just a hard threshold. A span that’s 60% Thai is probably a true entity in mixed text; one that’s 10% Thai is noise. Use the ratio as a confidence modifier, not a filter.

5. Add a real Thai date recognizer. DATE_TIME in the eval got 0% F1 — WangchanBERTa’s thainer-v2 dataset doesn’t have a DATE_TIME tag for Thai expressions like 15 มกราคม พ.ศ. 2567 or เมื่อวานนี้. A separate regex+lexicon recognizer for Thai Buddhist Era dates is needed.

6. Wrap the tokenizer-offset fix in a unit test. The <unk> drift bug cost 30 minutes to debug and only shows up on Thai text with rare characters. A pytest that runs the recognizer on สมชาย ทำงานที่กรุงเทพ and asserts the LOCATION span lands at [14:21] would catch this regression instantly.

7. Validate on a real production dataset, not a 60-case test. The 66% F1 is on a hand-picked test set. A 1,000-sentence labeled production sample would give a real number. The test cases here are for development, not for shipping.

8. GPU on the inference side. CPU inference is ~50ms per sentence. GPU is ~5ms. For a real-time de-identification service, that’s the difference between a synchronous pipeline and a batched one.

9. Persist the analyzer between requests. Building the pipeline (loading WangchanBERTa) takes ~6 seconds on first call. Use a global singleton, not per-request construction.

10. Add monitoring on the “no-PII” control sentence. The single most important regression test: every commit should re-run the analyzer on วันนี้อากาศดี ท้องฟ้าแจ่มใส เหมาะกับการออกไปเดินเล่น and assert zero detections. If your pipeline starts tagging weather sentences as PII, you’ve over-filtered the NER model’s output and your precision will silently degrade.

Files in this experiment

presidio/
├── wangchanberta_recognizer.py   # The Presidio-WangchanBERTa bridge (~250 lines)
├── thai_national_id.py           # Checksum-validated 13-digit ID recognizer
├── test_pipeline.py              # 5-case demonstration
├── test_corpus_50.py             # 60-case test corpus with auto-computed offsets
├── evaluate_50.py                # Quantitative P/R/F1 eval (permissive + strict)
├── evaluate.py                   # Old 5-case eval (kept for diff)
├── pipeline_output.txt           # Captured run output
├── eval_50_results.txt           # Permissive-mode metrics
├── eval_50_strict_results.txt    # Strict-mode metrics (drops bad-checksum IDs)
└── test_ner_only.py, test_thai.py, test_presidio.py  # Earlier exploration

All files run inside a Python 3.13 venv (uv venv .venv) with presidio-analyzer, presidio-anonymizer, pythainlp, and transformers installed. WangchanBERTa is auto-downloaded on first run (~250MB).

Bottom line

Wiring Presidio to WangchanBERTa is a Saturday-afternoon project, not a research project. The hard part isn’t the integration — it’s the post-filtering, the corpus quality, and the test data. The pipeline is good enough to use as a first-pass PII filter for Thai text in a real system, with the caveat that a human review step should be downstream of it for free-text entities (PERSON, ORG, LOCATION). For structured entities (email, phone, national ID, money, dates, URLs), it’s production-grade.

The two bugs that ate the most time were:

1. Tokenizer drift from <unk> tokens — adds 4 chars per OOV char to your offset table, and the failure is silent (off-by-N spans) until something crashes
2. Deduplication tax from English recognizers firing on Thai text — every email triggered a URL match, every Thai phone triggered a PHONE_NUMBER match, every Thai national ID triggered US_BANK_NUMBER and US_DRIVER_LICENSE matches

Both are addressed in the final code. Both would have been showstoppers in production without the fixes.

If you ship this, the most important thing to monitor is the false-positive rate on non-PII text — case 4 in the test corpus, the weather sentence. If your pipeline starts tagging weather sentences as PII, you’ve over-filtered the NER model’s output and your precision will drop below your recall. That’s the failure mode that will silently leak real PII through the cracks while everyone focuses on the false-negative rate.

Build the no-PII test case into your CI. Run it on every recognizer change. The pipeline is only as good as its worst day.