zyocum / dedupLinks
Find duplicate text files.
☆15Updated last year
Alternatives and similar repositories for dedup
Users that are interested in dedup are comparing it to the libraries listed below
Sorting:
- Crawling engine that crawls a set of top-level domains looking for documents in a list of languages☆11Updated last year
- Scripts for building a geo-located web corpus using Common Crawl data☆11Updated 2 weeks ago
- An example of how to use spaCy for extremely large files without running into memory issues☆36Updated 3 years ago
- Post-processing OCR errors with seq2seq models☆28Updated 5 years ago
- Next-generation Punkt sentence boundary detection with zero dependencies☆27Updated 2 months ago
- This is a prototype of a multi-lingual suite for named-entity recognition in Python.☆21Updated last year
- Convert CoNLL output of a dependency parser into a latex or graphviz tree☆12Updated 5 years ago
- Extract dates from text☆66Updated 5 years ago
- Correction of spaces with character-based neural language models.☆13Updated 3 years ago
- ☆20Updated 4 years ago
- This repo contains the code used to generate the French Wikipedia sample used in the QA annotation project PIAF☆11Updated 4 years ago
- A python library to generate highly realistic typos (fuzz-testing)☆13Updated 10 months ago
- ☆30Updated 3 years ago
- Tool for sentiment analysis annotation☆13Updated 10 months ago
- Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …☆62Updated 5 years ago
- Transformer based Trigram Blocking implementation in Tensorflow☆11Updated 5 years ago
- BERT models for many languages created from Wikipedia texts☆33Updated 5 years ago
- Text preprocessing tools in python.☆27Updated 7 years ago
- Dataiku DSS plugin to detect languages, correct misspellings, and clean text data 🧼☆22Updated this week
- Find strings/words in text; convenience and C speed☆126Updated 3 years ago
- The projects lets you extract glossary words and their definitions from a given piece of text automatically using NLP techniques☆30Updated 5 years ago
- This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…☆34Updated 2 years ago
- 📄Neural Sentential Paraphrase Generation to Augment Chatbot Training Dataset☆21Updated 3 years ago
- Annotation Management for Prodigy, that support multiple users working in many projects☆15Updated 7 years ago
- Analyze XML extracted from PDFs (e.g. from TET or PDFMiner)☆20Updated 8 years ago
- Fastlaw's purpose is to replace generic word embeddings for work on supervised machine learning NLP-tasks with legal texts.☆40Updated 6 years ago
- Loan Risk Prediction Neural Network and API☆17Updated 5 years ago
- Python wrapper for a C++ Double Metaphone☆15Updated 3 weeks ago
- A PyPI package for easy text annotation in a Jupyter Notebook.☆29Updated 4 years ago
- Multi-Langauge Identification☆28Updated last year