spotify / luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
☆18,024Updated this week
Alternatives and similar repositories for luigi:
Users that are interested in luigi are comparing it to the libraries listed below
- Apache Airflow - A platform to programmatically author, schedule, and monitor workflows☆38,298Updated this week
- Parallel computing with task scheduling☆12,851Updated this week
- A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C proje…☆22,902Updated this week
- Data-Centric Pipelines and Data Versioning☆6,197Updated this week
- Python composable command line interface toolkit☆15,961Updated this week
- An orchestration platform for the development, production, and observation of data assets.☆12,300Updated this week
- Always know what to expect from your data.☆10,117Updated this week
- Embrace the APIs of the future. Hug aims to make developing APIs as simple as possible, but no simpler.☆6,869Updated 6 months ago
- A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin☆6,241Updated last month
- Open Source AI/ML Platform☆8,446Updated this week
- Prefect is a workflow orchestration framework for building resilient data pipelines in Python.☆18,036Updated this week
- A next-generation curated knowledge sharing platform for data scientists and other technical professions.☆5,499Updated 4 months ago
- Hypothesis is a powerful, flexible, and easy to use library for property-based testing.☆7,674Updated this week
- Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.☆27,335Updated 2 weeks ago
- Python packaging and dependency management made easy☆32,328Updated this week
- the portable Python dataframe library☆5,451Updated this week
- Python Stream Processing☆6,757Updated 5 months ago
- 🦉 Data Versioning and ML Experiments☆14,088Updated this week
- Declarative visualization library for Python☆9,518Updated this week
- Open source platform for the machine learning lifecycle☆19,253Updated this week
- Modin: Scale your Pandas workflows by changing a single line of code☆9,975Updated last week
- Retrying library for Python☆6,936Updated 2 months ago
- Python datetimes made easy☆6,328Updated this week
- Data Apps & Dashboards for Python. No JavaScript Required.☆21,802Updated this week
- 💫 Industrial-strength Natural Language Processing (NLP) in Python☆30,670Updated this week
- Grumpy is a Python to Go source code transcompiler and runtime.☆10,544Updated 2 years ago
- Python Development Workflow for Humans.☆24,964Updated this week
- Accelerate your web app development | Build fast. Run fast.☆18,203Updated this week
- Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per s…☆8,321Updated 3 months ago
- Cookiecutter template for a Python package.☆4,287Updated last month