riotu-lab / aranizer
Aranizer: A Custom Tokenizer based on SentencePiece and BPE tailored for Arabic Language Modeling
☆16Updated 6 months ago
Alternatives and similar repositories for aranizer:
Users that are interested in aranizer are comparing it to the libraries listed below
- Instruction dataset for Arabic with 10,000 instruction and output pairs. CIDAR can be used to fine-tune LLMs to follow instructions.☆34Updated 11 months ago
- Python intefrace for evaluation on chatgpt models☆19Updated last year
- This is the official repository for Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks.☆24Updated 2 months ago
- أسئلة باللغة العربية تركز على الثقافة السعودية تم اختبارها على عدد من النماذج اللغوية الضخمة LLMs☆13Updated 3 weeks ago
- Arabic nested named entity recognition☆33Updated 9 months ago
- TURJUMAN, a neural toolkit for translating from 20 languages into Modern Standard Arabic (MSA).☆52Updated last year
- Seq2Seq-based open domain empathetic conversational model for Arabic: Dataset & Model☆57Updated 7 months ago
- ☆120Updated 11 months ago
- Arabic edition of ALBERT pretrained language models☆16Updated 3 years ago
- This is a diacritization model for Arabic language. This model was built/trained using the Tashkeela: the Arabic diacritization corpus on…☆41Updated last year
- ANETAC: Arabic Named Entity Transliteration and Classification Dataset☆34Updated 5 years ago
- The official implementation of CATT Arabic diacritization models.☆39Updated last month
- Arabic cleaning, normalization and segmentation library.☆66Updated last year
- ☆9Updated 4 years ago
- Benchmark Arabic text diacritization dataset☆73Updated 5 years ago
- hULMonA (حلمنا): tHe first Universal Language MOdel iN Arabic☆46Updated 4 years ago
- A Python implementation of Farasa toolkit☆122Updated 4 months ago
- A simple semi-supervised approach for creating huggingface data script loaders and upload to the hub.☆11Updated 7 months ago
- ☆41Updated 2 years ago
- A framework for Arabic spelling correction using different seq2seq model architectures such as transformers and RNNs☆20Updated 7 months ago
- Context-Sensitive Neural Spelling Checker☆20Updated 4 months ago
- Code, models, and data for "Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation". EMNLP 2023.☆16Updated 5 months ago
- Code and models for "The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models". EACL 2021, WANLP.☆45Updated 8 months ago
- AraT5: Text-to-Text Transformers for Arabic Language Understanding☆88Updated 9 months ago
- ArSarcasm-v2 is an extension to the original ArSarcasm dataset. It was used for the shared task on sarcasm detection and sentiment analys…☆11Updated 3 years ago
- A Docker image for a relatively light-weight full Arabic speech synthesis system☆29Updated 4 years ago
- The first Dialectal Arabic Code Switching - DACS corpus from broadcast speech. Annotated at the token-level, considering both the linguis…☆14Updated 2 years ago
- Multi-Modal Language Modeling with Image, Audio and Text Integration, included multi-images and multi-audio in a single multiturn.☆17Updated last year
- ☆12Updated 4 months ago
- UBC ARBERT and MARBERT Deep Bidirectional Transformers for Arabic☆104Updated 3 years ago