Languages
Learnarr works with any language your subtitles are in: suggestions, cards, the transcript and studying work for all of them. Two things depend on language data:
- "% you know" needs a lemmatiser model, which matches every word to its dictionary form.
- How common a word is needs a frequency list.
What each language has
52 of 63 languages have "% you know"; 50 have a frequency list.
| Language | "% you know" | Frequency list |
|---|---|---|
Afrikaans af | ✓ (afribooms) | ✓ |
Albanian sq | — | ✓ |
Arabic ar | ✓ (padt) | ✓ |
Armenian hy | ✓ (armtdp) | ✓ |
Basque eu | ✓ (bdt) | ✓ |
Belarusian be | ✓ (hse) | — |
Bengali bn | — | ✓ |
Bosnian bs | — | ✓ |
Bulgarian bg | ✓ (btb) | ✓ |
Catalan ca | ✓ (ancora) | ✓ |
Chinese zh | ✓ (gsdsimp) | ✓ |
Croatian hr | ✓ (set) | ✓ |
Czech cs | ✓ (pdt) | ✓ |
Danish da | ✓ (ddt) | ✓ |
Dutch nl | ✓ (alpino) | ✓ |
English en | ✓ (ewt) | ✓ |
Estonian et | ✓ (edt) | ✓ |
Finnish fi | ✓ (tdt) | ✓ |
French fr | ✓ (gsd) | ✓ |
Galician gl | ✓ (ctg) | ✓ |
Georgian ka | — | — |
German de | ✓ (hdt) | ✓ |
Greek el | ✓ (gdt) | ✓ |
Hebrew he | ✓ (htb) | ✓ |
Hindi hi | ✓ (hdtb) | ✓ |
Hungarian hu | ✓ (szeged) | ✓ |
Icelandic is | — | ✓ |
Indonesian id | ✓ (gsd) | ✓ |
Irish ga | ✓ (idt) | — |
Italian it | ✓ (isdt) | ✓ |
Japanese ja | ✓ (gsd) | — |
Kazakh kk | — | — |
Korean ko | — | ✓ |
Latvian lv | ✓ (lvtb) | ✓ |
Lithuanian lt | ✓ (alksnis) | ✓ |
Macedonian mk | — | ✓ |
Malay ms | — | ✓ |
Maltese mt | ✓ (mudt) | — |
Marathi mr | ✓ (ufal) | — |
Northern Sami se | ✓ (giella) | — |
Norwegian no | ✓ (bokmaal) | ✓ |
Norwegian Nynorsk nn | ✓ (nynorsk) | — |
Persian fa | ✓ (seraji) | ✓ |
Polish pl | ✓ (pdb) | ✓ |
Portuguese pt | ✓ (bosque) | ✓ |
Romanian ro | ✓ (rrt) | ✓ |
Russian ru | ✓ (syntagrus) | ✓ |
Scottish Gaelic gd | ✓ (arcosg) | — |
Serbian sr | ✓ (set) | ✓ |
Slovak sk | ✓ (snk) | ✓ |
Slovenian sl | ✓ (ssj) | ✓ |
Spanish es | ✓ (ancora) | ✓ |
Swedish sv | ✓ (talbanken) | ✓ |
Tagalog tl | — | ✓ |
Tamil ta | ✓ (ttb) | ✓ |
Telugu te | ✓ (mtg) | ✓ |
Thai th | — | — |
Turkish tr | ✓ (imst) | ✓ |
Ukrainian uk | ✓ (iu) | ✓ |
Urdu ur | ✓ (udtb) | ✓ |
Uyghur ug | ✓ (udt) | — |
Vietnamese vi | ✓ (vtb) | ✓ |
Wolof wo | ✓ (wtb) | — |
A language with neither still works: its words are split on spaces, and there's no score.
Lemmatiser models
Learnarr uses UDPipe with the Universal Dependencies 2.5 models from LINDAT/CLARIAH-CZ. They aren't part of the Docker image: each server downloads the one it needs the first time someone learns a language (15–60 MB), into /config/udpipe/models. Every download is checked against a checksum taken from a real download, and thrown away if it doesn't match.
To download a model ahead of time, or try again after a failed download, use System → Status → Languages.
Where a language has several treebanks, Learnarr picks the one with the best lemmas and the most everyday text (German HDT, English EWT, Chinese GSDSimp for simplified characters). Korean is left out: its models give morpheme chains rather than dictionary forms. Japanese and Chinese are written without spaces; UDPipe splits them into words.
The models are licensed CC BY-NC-SA 4.0: fine for a free, self-hosted app.
Frequency lists
The 10,000 most common words of each language, from Hermit Dave's FrequencyWords, counted over film and TV subtitles from OpenSubtitles (CC BY-SA 4.0). They're part of Learnarr itself.
Spotted a wrong lemma?
The models are trained on written text, so slang, dialect and very short lines trip them up. Mark a misread word known and it's fixed everywhere. If a language looks wrong more often than that, open an issue with a few examples: another treebank may do better.