NLP-ресурсы азербайджанского языка
Открытый реестр NLP-ресурсов азербайджанского языка — корпусы, датасеты, бенчмарки, модели и инструменты.
Корпусы (9)
DOLLMA
Primary LLM pretraining corpus
CC-100-AZ
Common Crawl subset for Azerbaijani
OSCAR-AZ
OSCAR Azerbaijani subset
Wikipedia-AZ
Azerbaijani Wikipedia dump
mC4-AZ
Multilingual C4 Azerbaijani subset
CulturaX-AZ
CulturaX Azerbaijani subset
HPLT-AZ
HPLT v1.0 Azerbaijani
FineWeb-AZ
FineWeb Azerbaijani subset
Glot500-AZ
Glot500 Azerbaijani subset
Датасеты (16)
AzNER
Named entity recognition dataset
AzSentiment
Sentiment analysis dataset
AzQA
Question answering dataset
AzParaphrase
Paraphrase detection dataset
TurkicMT-AZ
Turkic cross-lingual MT benchmark
FLORES-AZ
FLORES-200 Azerbaijani subset
NLLB-Seed-AZ
NLLB seed data for Azerbaijani
XL-Sum-AZ
XL-Sum Azerbaijani subset
Belebele-AZ
Belebele Azerbaijani subset
SIB-200-AZ
SIB-200 Azerbaijani subset
660K-AZ-EN-Parallel
Largest public Az-En parallel corpus; includes reliability scores and source tracking
Orca-AZ
Azerbaijani translation of Microsoft Orca instruction dataset
AzCon
Azerbaijani conversational QA dataset
UltraFeedback-AZ
First Azerbaijani preference dataset for RLHF/DPO alignment
Alpaca-AZ-Cleaned
Azerbaijani translation of Stanford Alpaca cleaned dataset
AZ-EN-Dataset
Az-En parallel translation dataset
Модели (10)
BERT-AZ
Azerbaijani BERT base model
mBERT
Multilingual BERT with Az support
XLM-R-AZ
XLM-RoBERTa with Az support
NLLB-AZ
NLLB translation model with Az
mT5-AZ
Multilingual T5 with Az support
Qwen2-AZ
Qwen2 with Azerbaijani capability
Llama3-AZ
Llama 3 with Azerbaijani
GPT-AZ
Azerbaijani GPT model
AzerBERT
AzerBERT language model
TURNA-AZ
Turkish-Azerbaijani transfer model