Research
Standard idiom datasets do one of two things: decide whether a sentence is idiomatic, or locate the idiom. MultiIdiom does both for every occurrence, which is what a tool needs if it is going to mark up running text rather than classify it.
An idiom is non-compositional: its meaning cannot be derived from its parts. That single property defeats word-indexed lookup and literal translation alike, and it makes idioms a persistent failure case for both language learners and NLP systems.
It is also occurrence-dependent. Spilled the beans is idiomatic in one sentence and literal in the next, so a resource that labels phrases rather than usages cannot tell you which one you are reading. Detection has to happen per occurrence, in context, with the extent of the idiom marked — otherwise a downstream tool has nothing to highlight.
Each record carries the idiom, a sense number, one or more definitions, register and region tags, and usage examples. Every example is labelled idiomatic or literal and carries the character-level start and end offsets of the idiom within the sentence.
Roughly a fifth of the annotated senses are labelled literal. They are published deliberately: a model that only ever sees idiomatic usage cannot learn to tell the two apart, and the literal cases are what make the idiomaticity label meaningful rather than a constant.
MultiIdiom was constructed with a six-stage LLM-assisted pipeline, seeded from Wiktionary idiom entries and validated against human judgements on a sample of each language.
Stage-by-stage description to be added from the paper draft.
English, Spanish, Hindi and Telugu are split 80/10/10 at the idiom-type level, so no idiom appears in more than one split. Indonesian, French and German are held out entirely as zero-shot test languages.
| Split | EN | ES | HI | TE | ID | FR | DE | Total |
|---|---|---|---|---|---|---|---|---|
| Train | 2,030 | 2,034 | 508 | 506 | — | — | — | 5,078 |
| Dev | 252 | 254 | 62 | 61 | — | — | — | 629 |
| Test | 251 | 254 | 62 | 62 | 325 | 800 | 800 | 2,554 |
| Total | 2,533 | 2,542 | 632 | 629 | 325 | 800 | 800 | 8,261 |
Idiomaticity labels were checked against independent human judgements for two languages.
Three models, three different relationships to MultiIdiom and to what’s actually running. Not all of them are the same checkpoint they might look like at a glance.
One forward pass yields a literal/idiomatic label and the span indices. Task heads ship separately in task_heads.pt — see the model card for the exact loading procedure. Powers the extractor’s "Literal vs. idiomatic" method — the only one that judges sentences with no idiom at all, not just the ones with one.
A LoRA adapter over mBERT, fine-tuned on the public ID10M dataset — predates MultiIdiom and is not the model the research below evaluates. It is the default "Model" method in the extractor today because it is the only full BIO checkpoint that still exists as deployable weights. Structurally has no literal/idiomatic decision: it flags a span or it doesn’t.
Full fine-tune (not LoRA), trained and evaluated on MultiIdiom — this is the checkpoint the accuracy figures on this site and in the paper refer to. Not currently deployed: the trained weights exist only as saved predictions, not as a loadable checkpoint, so it cannot serve the live extractor until retrained or recovered.
| Language | Code | Status | Dictionary idioms | Test instances |
|---|---|---|---|---|
| English | en | trained | 9,643 | 251 |
| Spanish | es | trained | 3,187 | 254 |
| Hindi | hi | trained | 183 | 62 |
| Telugu | te | trained | 103 | 62 |
| German | de | zero-shot | 1,018 | 800 |
| French | fr | zero-shot | 332 | 800 |
| Indonesian | id | zero-shot | 46 | 325 |
Citation details and BibTeX are on the publications page. The tools built on this work are the extractor and the dictionary.