NLP resources

Azerbaijani NLP resources

A curated, open roster of Azerbaijani NLP resources — corpora, datasets, benchmarks, models and tools.

Corpora (9)

DOLLMA

651M words · CC-BY-SA

Primary LLM pretraining corpus

CC-100-AZ

29M sentences · unknown

Common Crawl subset for Azerbaijani

OSCAR-AZ

1.2GB · CC0

OSCAR Azerbaijani subset

Wikipedia-AZ

190K articles · CC-BY-SA

Azerbaijani Wikipedia dump

mC4-AZ

4.5GB · ODC-BY

Multilingual C4 Azerbaijani subset

CulturaX-AZ

2.1GB · CC-BY-SA

CulturaX Azerbaijani subset

HPLT-AZ

800MB · CC0

HPLT v1.0 Azerbaijani

FineWeb-AZ

500MB · ODC-BY

FineWeb Azerbaijani subset

Glot500-AZ

150MB · Apache-2.0

Glot500 Azerbaijani subset

Datasets (16)

AzNER

15K samples · MIT

Named entity recognition dataset

AzSentiment

5K samples · MIT

Sentiment analysis dataset

AzQA

10K samples · CC-BY

Question answering dataset

AzParaphrase

8K pairs · CC-BY

Paraphrase detection dataset

TurkicMT-AZ

50K pairs · CC-BY

Turkic cross-lingual MT benchmark

FLORES-AZ

2K sentences · CC-BY-SA

FLORES-200 Azerbaijani subset

NLLB-Seed-AZ

6K sentences · CC-BY

NLLB seed data for Azerbaijani

XL-Sum-AZ

10K samples · CC-BY-NC-SA

XL-Sum Azerbaijani subset

Belebele-AZ

900 samples · CC-BY-SA

Belebele Azerbaijani subset

SIB-200-AZ

200 samples · CC-BY-SA

SIB-200 Azerbaijani subset

660K-AZ-EN-Parallel

666K pairs · unknown

Largest public Az-En parallel corpus; includes reliability scores and source tracking

Orca-AZ

500K samples · unknown

Azerbaijani translation of Microsoft Orca instruction dataset

AzCon

237K samples · unknown

Azerbaijani conversational QA dataset

UltraFeedback-AZ

61K pairs · unknown

First Azerbaijani preference dataset for RLHF/DPO alignment

Alpaca-AZ-Cleaned

52K samples · unknown

Azerbaijani translation of Stanford Alpaca cleaned dataset

AZ-EN-Dataset

549K pairs · unknown

Az-En parallel translation dataset

Benchmarks (4)

TUMLU

multi-task · MIT

Turkic Unified Multilingual Language Understanding benchmark

AzEval

multi-task · CC-BY

Comprehensive Azerbaijani evaluation suite

XTREME-AZ

multi-task · Apache-2.0

XTREME Azerbaijani tasks

XNLI-AZ

5K samples · CC-BY-NC

Cross-lingual NLI Azerbaijani

Models (10)

BERT-AZ

110M params · Apache-2.0

Azerbaijani BERT base model

mBERT

110M params · Apache-2.0

Multilingual BERT with Az support

XLM-R-AZ

270M params · MIT

XLM-RoBERTa with Az support

NLLB-AZ

1.3B params · CC-BY-NC

NLLB translation model with Az

mT5-AZ

300M params · Apache-2.0

Multilingual T5 with Az support

Qwen2-AZ

7B params · Apache-2.0

Qwen2 with Azerbaijani capability

Llama3-AZ

8B params · Llama-3

Llama 3 with Azerbaijani

GPT-AZ

1.5B params · CC-BY-NC

Azerbaijani GPT model

AzerBERT

110M params · MIT

AzerBERT language model

TURNA-AZ

400M params · Apache-2.0

Turkish-Azerbaijani transfer model

Tools (3)

awesome-azerbaijani-nlp

resource list · CC0

Curated list of Az NLP resources

AzMorph

morphological analyzer · MIT

Azerbaijani morphological analyzer

AzSpell

spell checker · MIT

Azerbaijani spell checker