Thai researchers adapt Ai2’s Dolma toolkit to build Mangosteen 47-billion-token corpus

Ai2 released Dolma as an open-source toolkit for large-scale data curation, providing not only finished datasets but also the pipeline for filtering, deduplication and quality control. A team led by Wannaphong Phatthiyaphaibun, a doctoral candidate at the Vidyasirimedhi Institute of Science and Technology (VISTEC), used Dolma to create a Thai-specific corpus called Mangosteen, comprising 47 billion tokens, and demonstrated that a smaller, well-filtered dataset can outperform larger ones.

Publicly available Thai corpora are largely built from web crawls without native speaker validation. The researchers found unsuitable content for training and missed high-quality sources such as books, academic papers, official websites and YouTube subtitles. Building a curation pipeline from scratch would have been too large a project for a small team; Dolma gave them a starting point that let them focus on Thai-specific adjustments instead of reinventing the wheel.

The original pipeline assumed English-style sentence and paragraph boundaries. Thai script lacks spaces between words and uses different punctuation, so sentence-level deduplication removed almost all data. The team retained deduplication at the document and URL level, swapped the language-identification tool, fine-tuned quality filters and added rules for common Thai patterns—for example, news pages that contain only a fragment followed by “read more”, which required extra filtering to discard incomplete articles.

In experiments with Thai LLMs, the adapted pipeline discarded more than 80 % of the Common Crawl data and almost half of FineWeb2, a web corpus that had already been cleaned, while preserving or improving performance compared with models trained on the larger corpora. Gains were observed across model sizes, especially on Thai cultural knowledge benchmarks, which the team cites as evidence that locally relevant data helps models better represent the communities they serve.

“the open nature of Dolma allowed the team to examine, modify, test and reproduce results for their language,” Phatthiyaphaibun said. The open-source nature of the toolkit, combined with local expertise, enabled the creation of Mangosteen and shows how community-driven adaptation can produce data that closed, generic models would never reach.