spotify / luigiLinks
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
☆18,616Updated this week
Alternatives and similar repositories for luigi
Users that are interested in luigi are comparing it to the libraries listed below
Sorting:
- Apache Airflow - A platform to programmatically author, schedule, and monitor workflows☆43,803Updated this week
- Parallel computing with task scheduling☆13,695Updated this week
- A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin☆6,502Updated 2 months ago
- Prefect is a workflow orchestration framework for building resilient data pipelines in Python.☆21,253Updated this week
- A cross-platform command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, C proje…☆24,512Updated 3 weeks ago
- Data-Centric Pipelines and Data Versioning☆6,277Updated 11 months ago
- SQL for Humans™☆7,224Updated last year
- Docker Apache Airflow☆3,814Updated 2 years ago
- Python composable command line interface toolkit☆17,099Updated last week
- Python Stream Processing☆6,831Updated last year
- Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.☆28,127Updated last week
- Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.☆28,052Updated 2 months ago
- Distributed Task Queue (development branch)☆27,851Updated this week
- 📚 Parameterize, execute, and analyze notebooks☆6,349Updated this week
- Embrace the APIs of the future. Hug aims to make developing APIs as simple as possible, but no simpler.☆6,907Updated last year
- The no-magic web API and microservices framework for Python developers, with a focus on reliability and performance at scale.☆9,777Updated last week
- Simple job queues for Python☆10,539Updated this week
- Simple, Pythonic remote execution and deployment.☆15,352Updated 5 months ago
- A curated list of awesome ETL frameworks, libraries, and software.☆3,508Updated last year
- 🦉 Data Versioning and ML Experiments☆15,249Updated this week
- GraphQL framework for Python☆8,243Updated 4 months ago
- Build, Manage and Deploy AI/ML Systems☆9,702Updated this week
- Python datetimes made easy☆6,604Updated 3 weeks ago
- Curated list of resources about Apache Airflow☆3,868Updated last year
- Python Development Workflow for Humans.☆25,103Updated this week
- Serverless Python☆11,857Updated 2 years ago
- REST API framework designed for human beings☆6,744Updated last week
- 📘 The interactive computing suite for you! ✨☆6,276Updated 2 years ago
- An orchestration platform for the development, production, and observation of data assets.☆14,717Updated this week
- A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow☆2,085Updated 2 years ago