This dataset is derived from web-mined text (NLLB / CCMatrix, from Common Crawl). It is released under ODC-BY, but individual sentences may be protected by the copyright of their original publishers (a large share comes from religious websites such as jw.org). By requesting access you agree to: (1) use it for research and model development in accordance with the ODC-BY license and attribution requirements, (2) respect the Common Crawl terms of use and the rights of the original publishers, and (3) not redistribute the raw text as a standalone corpus without these notices.

NLLB Burkina Parallel (cleaned)

Cleaned sentence pairs between English, French, Mooré, Dyula, Bambara and Fulfulde, built from the NLLB mined bitext (allenai/nllb) for languages of Burkina Faso and neighbouring countries. It includes pairs that do not exist in NLLB, built by pivoting through English or French, notably French–Mooré.

588,640 pairs over 14 language pairs, one config per pair (plus all), train split only.
Cleaning: junk removal, untranslated copies and length-ratio filter, language identification with GlotLID, one-to-one deduplication (fixes NLLB's "hubness"), FLORES+ decontamination.
No alignment-score cutoff is applied: quality signals are provided as columns so you can choose your own thresholds (see Known limitations).

from datasets import load_dataset
ds = load_dataset("burkimbia/nllb-burkina-parallel", "fra-mos", split="train")

Config	Pairs	From NLLB file	Pivot	Notes
`eng-mos`	89,483	89,483	0
`eng-dyu`	56,924	56,924	0
`eng-bam`	43,248	43,248	0
`eng-fuv`	123,083	123,083	0	Fula, mixed dialects (see `trg_lid`)
`fra-mos`	18,223	0	18,223	via English (CCMatrix en–fr × NLLB en–mos)
`fra-dyu`	56,625	56,625	0
`fra-bam`	56,151	56,151	0
`fra-fuv`	96,537	96,537	0	Fula, mixed dialects
`mos-dyu`	5,183	0	5,183	via English
`mos-bam`	2,157	0	2,157	via English
`mos-fuv`	3,903	0	3,903	via English
`dyu-bam`	7,873	7,081	792	Manding pair
`dyu-fuv`	13,611	11,978	1,633
`bam-fuv`	15,639	14,923	716
Total	588,640	556,033	32,607

Per-step counts for every pair are in stats/summary.csv.

Columns

All configs share the same schema. src is always the first language of the config name (order: eng, fra, mos, dyu, bam, fuv).

Column	Type	Description
`id`	string	Stable id: first 16 hex chars of sha1(`src_lang`, `trg_lang`, `src_text`, `trg_text`, tab-joined)
`src_lang`, `trg_lang`	string	FLORES-200 codes: `eng_Latn`, `fra_Latn`, `mos_Latn`, `dyu_Latn`, `bam_Latn`, `fuv_Latn`
`src_text`, `trg_text`	string	Sentence pair (whitespace-normalised)
`src_url`, `trg_url`	string, nullable	Source web page, from the NLLB files. Null when unknown (e.g. the French side of `fra-mos`)
`src_lid`, `trg_lid`	string	GlotLID top-1 label. For Fula this is the predicted dialect (`fuh_Latn`, `ffm_Latn`, `fuv_Latn`, …)
`src_lid_prob`, `trg_lid_prob`	float	GlotLID top-1 probability
`length_ratio`	float	min(len) / max(len) of the two texts in characters (1 = same length)
`method`	string	`direct` (NLLB file), `pivot:eng`, `pivot:fra`, `pivot:eng(ccmatrix)`
`pivot_lang`, `pivot_text`	string, nullable	Bridge sentence for pivot pairs
`score`	float	Original NLLB / CCMatrix alignment (margin) score. Not calibrated, see below
`blaser`	float, nullable	Reserved for BLASER 2.0-QE scores (empty in this version)

How it was built

Sources. NLLB raw files from the AllenAI bucket (eng–mos, eng–dyu, eng–bam, eng–fuv, fra–dyu, fra–bam, fra–fuv, bam–dyu, dyu–fuv, bam–fuv) and the CCMatrix en–fr file (328.6M pairs, used only as a pivot). Scripts are in scripts/.

Pivot pairs. NLLB has no French–Mooré, Mooré–Dyula, Mooré–Bambara or Mooré–Fula data. These pairs are built by joining two files on the same English (or French) sentence, matched case-, punctuation- and whitespace-insensitively, with at most 5 candidates per bridge sentence. A pivot pair's score is the lower of its two alignment scores. Pivot rows are also added to dyu-bam, dyu-fuv and bam-fuv.

Cleaning, in order, applied to every pair:

Basic: drop empty, letter-less or very short sentences (< 3 alphanumeric characters).
Copies / length: drop untranslated copies (same text on both sides) and pairs with length_ratio < 1/3 when the longer side exceeds 20 characters; drop exact duplicates.
Language ID with GlotLID on both sides. English, French and Mooré must be detected as themselves. Bambara and Dyula accept each other, because they are closely related Manding languages that GlotLID cannot reliably separate. Fula accepts any Fula variety.
One-to-one: each sentence (by match key) is kept once per side, with its best-scoring partner. NLLB pairs a single Mooré sentence with up to 3,219 different English sentences ("hubness" of mined bitext), most of which are not translations.
FLORES+ decontamination: a pair is dropped if any of its sentences (including the pivot) equals, contains or is contained in a FLORES+ dev/devtest sentence of the same language (containment for ≥ 30 characters). No overlap was found (NLLB already excluded FLORES), so 0 pairs were removed. FLORES+ is safe to use for evaluation.

Known limitations

The score is not calibrated. It is a margin score from LASER (CCMatrix) or LASER3 (NLLB). Its scale differs by language pair, by model, and for pivot pairs, so a single threshold means different things in different configs. Use it to rank within a config. Manual inspection suggests many pairs in the 1.05–1.08 range are usable, but some are paraphrases or only topically related.
Fula dialects are predictions. NLLB has only one Fula code (fuv_Latn, "Nigerian Fulfulde"), but its data mixes several dialects. We trust GlotLID's per-sentence label (src_lid/trg_lid) as the dialect. In eng-fuv it is roughly fuv 34% · fub 22% · fuf 14% · ffm 13% · fuh 10.5% · fue 4% · fuq 2%. For Burkina Faso / Niger Fulfulde, filter on fuh_Latn (26,870 pairs), optionally adding Maasina ffm_Latn (31,737). Dialect-level labels are less accurate than language-level ones, especially on short sentences.
Noisy NLLB source. Before cleaning, only ~33% of NLLB's "Mooré" sentences were Mooré according to GlotLID; the rest included Roman Urdu, English, French, Hmong and Brazilian Indigenous languages. About half of the Dyula and Bambara sides were also other languages. Some noise remains; GlotLID is overconfident (lid_prob ≈ 1.0 for most rows), so the low-probability tail is the most useful signal.
Domain skew. A large share is religious text, especially Bible translations. jw.org alone accounts for 55,373 rows (≈ 39% of eng-mos, 46% of fra-mos). The rest is short, often informal web text.
Pivot quality. A pivot pair is only as good as its weaker link, and its two sides were never aligned directly.

License and attribution

Released under the Open Data Commons Attribution License (ODC-BY 1.0), the license of the source data. Access is gated because the underlying sentences come from web pages (via Common Crawl) whose publishers may hold copyright, notably religious publishers. You are also bound by the Common Crawl terms of use and the terms of the original sources. src_url/trg_url let you filter by source (e.g. trg_url NOT LIKE '%jw.org%').

Please cite the sources:

@article{nllb2022,
  title   = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  author  = {{NLLB Team} and Costa-juss{\`a}, Marta R. and others},
  journal = {arXiv preprint arXiv:2207.04672},
  year    = {2022}
}
@inproceedings{schwenk2021ccmatrix,
  title     = {{CCMatrix}: Mining Billions of High-Quality Parallel Sentences on the Web},
  author    = {Schwenk, Holger and Wenzek, Guillaume and Edunov, Sergey and Grave, Edouard and Joulin, Armand and Fan, Angela},
  booktitle = {Proceedings of ACL-IJCNLP},
  year      = {2021}
}
@inproceedings{kargaran2023glotlid,
  title     = {{GlotLID}: Language Identification for Low-Resource Languages},
  author    = {Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
  booktitle = {Findings of EMNLP},
  year      = {2023}
}

Maintainers

Burkimbia. Planned: BLASER 2.0-QE scores in the blaser column, a human-calibrated quality threshold, and a larger French–Mooré set.

Downloads last month: 12

Total file size:

69.9 MB

Paper for burkimbia/nllb-burkina-parallel

No Language Left Behind: Scaling Human-Centered Machine Translation

Paper • 2207.04672 • Published Jul 11, 2022 • 5

Access NLLB Burkina Parallel