spotify / luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
☆18,203Updated 2 months ago
Alternatives and similar repositories for luigi:
Users that are interested in luigi are comparing it to the libraries listed below
- Apache Airflow - A platform to programmatically author, schedule, and monitor workflows☆39,379Updated this week
- Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.☆27,500Updated this week
- Python composable command line interface toolkit☆16,201Updated 2 weeks ago
- Python Stream Processing☆6,777Updated 8 months ago
- A formatter for Python files☆13,866Updated 3 weeks ago
- Utils for streaming large files (S3, HDFS, gzip, bz2...)☆3,289Updated this week
- AWS SDK for Python☆9,269Updated this week
- A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin☆6,322Updated 3 weeks ago
- A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.☆9,874Updated last month
- Data-Centric Pipelines and Data Versioning☆6,214Updated last month
- Ready-to-run Docker images containing Jupyter applications☆8,156Updated this week
- Ultra fast asyncio event loop.☆10,788Updated 5 months ago
- Retrying library for Python☆7,238Updated last week
- Data Apps & Dashboards for Python. No JavaScript Required.☆22,225Updated this week
- Hypothesis is a powerful, flexible, and easy to use library for property-based testing.☆7,767Updated this week
- Grumpy is a Python to Go source code transcompiler and runtime.☆10,548Updated 3 years ago
- Simple job queues for Python☆10,093Updated last week
- Build, Manage and Deploy AI/ML Systems☆8,677Updated this week
- Embrace the APIs of the future. Hug aims to make developing APIs as simple as possible, but no simpler.☆6,879Updated 8 months ago
- Parallel computing with task scheduling☆13,083Updated last week
- Python job scheduling for humans.☆12,015Updated 10 months ago
- A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C proje…☆23,272Updated last week
- NumPy and Pandas interface to Big Data☆3,197Updated last year
- Accelerate your web app development | Build fast. Run fast.☆18,301Updated this week
- Modin: Scale your Pandas workflows by changing a single line of code☆10,095Updated this week
- Distributed Task Queue (development branch)☆25,937Updated this week
- Python packaging and dependency management made easy☆32,906Updated this week
- Computing with Python functions.☆4,022Updated last week
- Python client for Apache Kafka☆5,715Updated this week
- SQL for Humans™☆7,187Updated 8 months ago