Building and evaluating a Kyrgyz tokenizer
Abstract
I am studying how large language models work at every level, from the text that enters the system to the numbers and operations inside the network. Tokenization was one of the first parts I wanted to understand in detail. Instead of only reading about it, I decided to build a tokenizer for Kyrgyz and observe the full process myself.
The initial goal was educational: collect a real corpus, implement the main byte-pair merging logic, train a tokenizer, and measure how it represents Kyrgyz text. The first version used only Kyrgyz and worked well. I then extended the study to the mixed Kyrgyz-Russian language common in everyday life and trained 24 controlled bilingual candidates. The selected tokenizer has 32,768 tokens and uses a 90% Kyrgyz, 10% Russian training mix. Compared with the Kyrgyz-only version, it keeps 99.25% of the Kyrgyz compression, improves Russian compression by 35.33%, and improves mixed documents by 9.64%.
1. Introduction
My broader goal is to understand LLMs as systems rather than black boxes. That means studying each layer closely: how text becomes tokens, how tokens become vectors, how attention moves information, and how training changes the model's weights.
The tokenizer looked like a good place to begin because its input and output are easy to see. A person writes text, and the tokenizer turns it into a sequence of numbers. Yet small changes can produce different sequences. For example, OpenAI's tokenizer may split Hello World into two pieces while HELLO WORLD becomes three. Capitalization changes the result even though a human reads almost the same message.
I wanted to know who decides where these cuts appear, how common pieces are learned, and what happens to a language that has much less text online than English. I searched for practical work on Kyrgyz tokenization but found little material that documented the entire path from corpus construction to evaluation. Building my own version made the topic concrete and gave me a reason to work with a language that matters to me.
Kyrgyz and Russian do not stay in separate boxes in my daily life. I hear them mixed in the same conversations and see the same thing in messages, comments, and social media. Studies of Kyrgyz-Russian bilingualism also describe this code-switching as a regular feature of everyday speech and online comments. A tokenizer for real Kyrgyz usage has to expect mixed language.
The first research question was therefore simple: can I build a useful Kyrgyz tokenizer and understand every important step? I kept the first implementation Kyrgyz-only to give the experiment a clear starting point. Once it worked, the next question was whether it could handle the mixed language I already heard and saw every day.
2. Background: what a tokenizer does
A language model does not receive words directly. The tokenizer first cuts text into reusable pieces called tokens and assigns a number to each piece.
Imagine building sentences from Lego. If the box contains pieces that fit the language well, a sentence needs only a few of them. If the box was designed mostly for another language, the same sentence breaks into many small fragments. The resulting token sequence becomes longer, so less useful text fits into a fixed context window.
A tokenizer therefore balances two things. A small vocabulary uses less memory but usually produces longer sequences. A large vocabulary can store more complete words and word parts, but it also consumes more memory. The best size depends on the language and the intended use.
3. Method
3.1 Byte-level BPE
I used byte-level Byte Pair Encoding, usually shortened to byte-level BPE. The name is more complicated than the idea.
UTF-8 turns text into bytes. A byte can have one of 256 values, so these 256 possibilities form a universal starting vocabulary. Kyrgyz letters, Latin letters, punctuation, and emoji may use different numbers of bytes, but all of them can still be represented.
BPE then looks for the neighboring pair that appears most often and joins it into a new token. The algorithm counts again, makes another merge, and repeats until the vocabulary reaches the chosen size. If small pieces often form Кыр, the tokenizer may eventually learn Кыр as one token. Later it may combine that piece with гыз. These units come from frequency in the corpus, not from a handwritten dictionary.
Before BPE starts merging, I separate letters, numbers, punctuation, and spaces into categories. This prevents the algorithm from spending too many vocabulary slots on variants such as dog, dog!, and dog?. Merges can happen inside a fragment, but not freely across every category boundary.
КырD0 9AD1 8BD1 80КырКырКырSimplified example. The real merge order depends on pair frequencies in the corpus.
3.2 Corpus construction
The merging algorithm is short. Building a trustworthy Kyrgyz corpus took much more work.
I combined wide web text, Wikipedia, news, and the Manas epic. These sources contribute different writing styles, which reduces the risk of building a tokenizer around one narrow kind of text.
The final collection contains 188,208 documents and a little over 524 million UTF-8 bytes of Kyrgyz text.
Raw web text cannot simply be placed in one folder and called a corpus. Pages contain menus, advertisements, broken characters, copied news stories, and repeated sections. If duplicate material remains, BPE may treat repeated noise as an important language pattern.
The pipeline cleaned the text, checked the language, removed exact copies, and detected pages that were almost identical. This process removed 18,777 near-duplicate documents. I also recorded the source and license information. The repository does not redistribute the raw corpus because its sources have different terms.
3.3 From Kyrgyz-only to a bilingual experiment
The first tokenizer was trained only on Kyrgyz. It encoded the evaluated text without loss and compressed Kyrgyz efficiently. The first experiment had worked: I had built the training pipeline and produced a tokenizer that fit its target language.
The real world is less tidy than a Kyrgyz-only corpus. In Kyrgyzstan, especially in conversations and social media, people often move between Kyrgyz and Russian. A sentence can begin in Kyrgyz, use a Russian word or phrase, and then return to Kyrgyz. I hear and see this kind of mixed language every day.
Once the Kyrgyz version worked, I decided to run a second experiment. The question was how much Russian data the tokenizer needed to handle mixed language without sacrificing its Kyrgyz result. When I measured the first version on Russian and mixed text, Russian passages broke into too many small pieces.
I trained 24 candidates while changing three variables: the share of Russian in the corpus, the rule used to split text before BPE, and the vocabulary size. The tested vocabularies contained 32,000, 40,000, or 50,000 pieces.
Before running the comparison, I expected a 20% Russian share to work best. The results did not support that guess. A 10% share was enough to improve Russian sharply while preserving almost all of the Kyrgyz result. The 32K vocabulary was selected because the larger alternatives produced only small compression gains while requiring more memory.
4. Evaluation and results
The main metric was bytes per token. It answers a simple question: how much original text does one token carry? A higher value means that the same document becomes a shorter token sequence.
The bilingual tokenizer retained 99.25% of the Kyrgyz compression achieved by the Kyrgyz-only version. Russian compression improved by 35.33%, and mixed Kyrgyz-Russian documents improved by 9.64%.
I also tested 21 held-out documents containing both languages. Twenty became shorter with the bilingual tokenizer. One became longer by a single token. Every evaluated record decoded back into exactly the same text, so the shorter representation did not lose information.
The average result still hides small tradeoffs. A few short Kyrgyz sentences gained one token, and some Kyrgyz suffixes split less neatly because Russian pieces occupied part of the fixed vocabulary. I accepted that small loss in exchange for a much larger improvement on Russian and mixed text.
5. Relation to existing work
I did not design the method in isolation. I studied the OpenAI GPT-2 encoder, the SentencePiece paper, the GigaChat family paper, and the DeepSeek-V3 technical report. These sources helped me compare vocabulary sizes, byte-level representations, text-splitting rules, and multilingual design choices.
The implementation borrows public ideas, especially reversible byte encoding and category-aware text splitting. It does not claim to reproduce a private training recipe. The useful part of this project is that the corpus decisions, experiments, and final selection are documented for Kyrgyz.
6. Limitations
This experiment measures tokenization, not the quality of a complete language model. A shorter sequence is useful, but it does not by itself prove that a model will understand Kyrgyz better.
The mixed evaluation set contains 21 documents and mostly represents formal writing. It does not yet measure informal code-switching in speech, chat, or social media as directly as I would like. Source code and heavily multilingual conversations may also produce different results. English and code can be encoded because the tokenizer starts from bytes, but they were not optimization targets.
7. Conclusion
The project began as a way to understand one layer of an LLM by building it myself. It produced a working 32,768-token byte-level BPE tokenizer, a cleaned Kyrgyz corpus pipeline, and a controlled comparison of bilingual variants. The selected 90% Kyrgyz and 10% Russian mixture preserved the strong Kyrgyz result while making the tokenizer much better suited to the mixed-language mode common in everyday life.
The most useful outcome is that tokenization is no longer an abstract preprocessing step to me. I can follow text into UTF-8 bytes, inspect the learned merges, encode it into token IDs, and decode those IDs back into the original text. I may eventually use this tokenizer in a small LLM experiment. For now, the tokenizer and its evaluation are complete, and the full implementation is available in the project repository.