Tossana · Khmer image captioning

Why

Describing pictures in Khmer is hard for a machine

Today's captioning models were trained almost only on English. Khmer brings problems they never see.

No Khmer caption data

There is no large open set of photos with Khmer descriptions. Tossana starts from English captions translated to Khmer, with the test set checked by people.

No spaces between words

Khmer is written without spaces, so word-based scores do not fit. Tossana is scored on Khmer letter clusters, the same units Chakkhu and Sota use.

Seeing things that are not there

Small captioning models often name objects that are not in the photo. For someone who cannot see the picture, a wrong object is worse than a short caption.

Architecture · the plan

A small eye that speaks Khmer

About 33 million parameters: a pretrained ViT-S image encoder, a small decoder that writes Khmer clusters, a fast concept head and a guard against invented objects. It shares Sota's recipe: Khmer clusters as tokens, int8 ONNX, beam search.

Figure 1. Tossana overview: image patches, ViT encoder, concept head, caption decoder, beam search with hallucination guard

Figure 1. Overview (planned design). 16×16 patches → 12 ViT blocks → a concept head (object tags) and a 4-layer caption decoder → beam search that drops any caption naming an object the concept head does not see.

Why this design

Starts from a trained eye. The ViT-S encoder already knows images (ImageNet); only the Khmer writer is learned from scratch.
Whole clusters, not letters. The same 3,402 Khmer cluster tokens as Chakkhu and Sota, so ស្ត្រី is one token and all three share one engine.
Checks itself. A small concept head lists the objects it sees; captions that name anything else are dropped (planned, the main research idea).
Runs on the phone. Encoder and decoder are exported to int8 ONNX (36 MB), like Sota: about 0.2 s per photo on a laptop CPU.

Four ways to describe a picture

Design	On a phone, in Khmer?
Large vision-language modelQwen2.5-VL, LLaVA: billions of parameters	too big
Fine-tune GIT / BLIPEnglish tokenizers, 150M+	no Khmer tokens
CNN + LSTMclassic, small	weak captions
ViT-S + cluster decoder + guardour choice, ≈33M	small + Khmer

A large model is still useful: as a teacher that writes more Khmer captions for Tossana to learn from (see the roadmap).

Benchmark

How good is it?

Scored on 1,910 test photos no model trained on: 1,000 everyday photos (Flickr8k, captions translated to Khmer) and 910 Cambodian scenes (phonsobon's Khmer captions). CIDEr: higher is better, counted on Khmer letter clusters. ⚠️ Test captions are not checked by people yet, so treat these as progress numbers.

71.0CIDEr on 1,000 everyday photos

22.0CIDEr on 910 Cambodian scenes (harder: one long caption per photo)

36 MBint8 ONNX · 32.8M parameters

0.2 sper photo on a laptop CPU

Each version, same test photos (CIDEr)

More and better Khmer data lifts both tests. Exp 2 had almost no everyday photos.

Everyday photos (1,000)

Exp 2 · 19k captions

6.5

Exp 4 · + Flickr8k

66.1

Exp 5 · + phonsobon v2

71.0

Cambodian scenes never seen in training (691)

Exp 4

13.4

Exp 5 (published)

20.6

Bars scaled per test (everyday to 80, Cambodian to 25).

PyTorch vs ONNX (same 300 photos)

Version	Size	CIDEr	Same caption as PyTorch
PyTorchreference	131 MB	54.4	—
ONNX fp32	136 MB	53.4	94%
ONNX int8default, phone & web	36 MB	55.6	45%

int8 scores the same but picks different words about half the time (ឆ្កែពណ៌ត្នោតធំមួយ… vs ឆ្កែពណ៌ត្នោត…). fp32 matches PyTorch.

Real captions (example photos)

Photo	Tossana wrote
white dog on grass	ឆ្កែសកំពុងរត់កាត់ស្មៅ
monks on a path	ព្រះសង្ឃមួយក្រុមឈរលើផ្លូវកខ្វក់…
boats, Tonle Sap	ទូកមួយក្រុមដែលអណ្តែតលើផ្ទៃទឹក…
Bayon temple	ប្រាសាទអង្គរវត្ត… (right idea, wrong temple)
girl climbing a stilt house	ឆ្កែពណ៌ខ្មៅ… ("a dog": wrong)

All runs

Run	Training data	Machine	Everyday CIDEr	Cambodian CIDEr	Learned
Exp 1English check	Flickr8k (English)	Colab T4	65.9 (English)	—	pipeline works
Exp 2	phonsobon v1 + news · 19k	Mac M4 Pro	6.5	—	news teaches invented places
Exp 3	same, longer training	RTX 3060	—	—	more steps alone: no gain
Exp 4	+ Flickr8k, heritage (MADLAD) · 49k	RTX 3060	66.1	15.0	more data: no memorizing
Exp 5published	+ phonsobon v2 · 49k	RTX 3090	71.0	22.0	more Cambodian data helps

Everyday = the same 1,000 Flickr8k test photos for every Khmer run; Cambodian = the 910-photo phonsobon test (from Exp 4 on). Every number: model card. Training speed: Mac M4 Pro 85, RTX 3060 312, RTX 3090 510 photos per second.

Data

Where the Khmer captions come from

There was no open Khmer captioning set big enough, so Tossana combines Khmer sets shared by others with English sets translated to Khmer. None of the Khmer sets states a license yet: research use, with credit.

Training (49,053 captions)

phonsobon v1 + v2Khmer descriptions of Cambodian scenes; duplicate photos removed across all splits	2,157
Flickr8k → Khmereveryday photos, translated with MADLAD-400 (chosen over NLLB-200 by a 150-photo comparison)	29,723
Angkor carvings → KhmerDarayut's heritage descriptions, translated and shortened	769
News photo captionsrinabuoy; photo credits removed; training only, never scored	16,331

Testing (never trained on)

Flickr8k test, in Khmer1,000 photos × 5 machine-translated captions	1,000
phonsobon testCambodian scenes; a photo in any test split never reaches training	910
Checked by people2 captions per photo read and fixed by a Khmer speaker	next

Found while preparing: phonsobon v2 repeats some photos up to 11× across its own train and test splits, and v1 and v2 split the same photos differently. Tossana's rule: a photo belongs to its strictest split everywhere.

Roadmap

Where we are

One step at a time, each with a check before the next.

Done

Design & English check

The ≈33M design, then the same pipeline in English: CIDEr 65.9, int8 ONNX works.

Done

Khmer data

49k captions: phonsobon v1 + v2, Flickr8k and Angkor carvings translated with MADLAD-400, news.

Done

Train & publish

Exp 5 on one RTX 3090 (27 min). PyTorch, ONNX and demo are live.

Now

Checked test set

A Khmer speaker reads and fixes 2 captions for each test photo, so the scores become final.

Guard against wrong objects

A concept head lists what it sees; captions that name anything else are dropped. The main research idea.

Later

In the apps

A browser demo like Sota's, and describing photos together with Chakkhu, offline.

Akkhara អក្ខរៈtypes: Sing Khmer → Khmer Chakkhu ចក្ខុreads: Khmer OCR Sota សោតៈlistens: Khmer ASR Tossana ទស្សនៈsees: Khmer captions VoxPocketspeaks: Khmer TTS

Citation

Using Tossana? Please cite it

@misc{ing2026tossana,
  author    = {Ing, Muyleang},
  title     = {Tossana: A Small Khmer Image Captioning Model},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/muyleanging/Tossana-Caption-Khmer}
}

Preview release: personal use, evaluation and research. The final license comes once the data licenses are confirmed.

Data, models and methods Tossana builds on

Source	License
khmer_Images_Captioning + v2phonsobon · Khmer descriptions of Cambodian scenes	not stated
Khmer-Captioningrinabuoy · news photo captions (training only)	not stated
khmer-historical-heritage-image-captioningDarayut · Angkor carvings (English, translated)	not stated
Flickr8kphotos and English captions · Hodosh, Young & Hockenmaier, JAIR 2013	research only
ViT-S/16pretrained image encoder · Dosovitskiy et al., ICLR 2021	Apache-2.0
NLLB-200English → Khmer translation (compared, not used) · NLLB Team, 2022	CC BY-NC 4.0
MADLAD-400English → Khmer translation (used) · Kudugunta et al., 2023	Apache-2.0
CIDEr · BLEUcaption scores · Vedantam et al., CVPR 2015 · Papineni et al., ACL 2002	method

Text normalization (khnormal) and the 3,402 Khmer cluster tokens are shared with Chakkhu and Sota.

BibTeX for these sources

@article{hodosh2013framing,
  title   = {Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics},
  author  = {Hodosh, Micah and Young, Peter and Hockenmaier, Julia},
  journal = {Journal of Artificial Intelligence Research},
  volume  = {47},
  pages   = {853--899},
  year    = {2013}
}

@inproceedings{dosovitskiy2021vit,
  title     = {An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author    = {Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2021}
}

@article{nllb2022,
  title   = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  author  = {{NLLB Team} and Costa-juss{\`a}, Marta R. and others},
  journal = {arXiv preprint arXiv:2207.04672},
  year    = {2022}
}

@article{kudugunta2023madlad,
  title   = {MADLAD-400: A Multilingual And Document-Level Large Audited Dataset},
  author  = {Kudugunta, Sneha and Caswell, Isaac and Zhang, Biao and Garcia, Xavier and Choquette-Choo, Christopher A. and Lee, Katherine and Xin, Derrick and Kusupati, Aditya and Stella, Romi and Bapna, Ankur and Firat, Orhan},
  journal = {arXiv preprint arXiv:2309.04662},
  year    = {2023}
}

@inproceedings{vedantam2015cider,
  title     = {CIDEr: Consensus-based Image Description Evaluation},
  author    = {Vedantam, Ramakrishna and Zitnick, C. Lawrence and Parikh, Devi},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages     = {4566--4575},
  year      = {2015}
}

Show a picture.
Read it in ខ្មែរ.

Show Tossana a photo

Describing pictures in Khmer is hard for a machine

No Khmer caption data

No spaces between words

Seeing things that are not there

A small eye that speaks Khmer

Why this design

Four ways to describe a picture

How good is it?

Each version, same test photos (CIDEr)

PyTorch vs ONNX (same 300 photos)

Real captions (example photos)

All runs

Where the Khmer captions come from

Training (49,053 captions)

Testing (never trained on)

Where we are

Design & English check

Khmer data

Train & publish

Checked test set

Guard against wrong objects

In the apps

Using Tossana? Please cite it

Data, models and methods Tossana builds on