furqanshahid85-python / Python-FTP-File-IngestionLinks
This module provides the functionality of uploading files to s3 from a FTP server. An SFTP connection is created with the FTP server and all the files present in the specified directory are uploaded to the specified s3 bucket. Following are the key features of this module: Creates a secure ssh connection with FTP server. Handles multipart upload…
☆13Updated 5 years ago
Alternatives and similar repositories for Python-FTP-File-Ingestion
Users that are interested in Python-FTP-File-Ingestion are comparing it to the libraries listed below
Sorting:
- (project & tutorial) dag pipeline tests + ci/cd setup☆90Updated 4 years ago
- Example DAGs using hooks and operators from Airflow Plugins☆348Updated 7 years ago
- A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract d…☆24Updated 4 years ago
- This is the documentation for the Amazon Redshift Developer Guide☆121Updated 2 years ago
- Move Data From Salesforce -> S3 -> Redshift☆33Updated 4 years ago
- Productionalizing Data Pipelines with Apache Airflow☆115Updated 3 years ago
- This project helps me to understand the core concepts of Apache Airflow. I have created custom operators to perform tasks such as staging…☆97Updated 6 years ago
- locopy: Loading/Unloading to Redshift and Snowflake using Python.☆115Updated last week
- A Python API for Asynchronously Loading Data into Snowflake DB -☆68Updated 3 months ago
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Updated 4 years ago
- Demo code to illustrate the execution of PyTest unit test cases for AWS Glue jobs in AWS CodePipeline using AWS CodeBuild projects☆49Updated last year
- Developed an ETL pipeline for a Data Lake that extracts data from S3, processes the data using Spark, and loads the data back into S3 as …☆16Updated 6 years ago
- Sample pytest tests for testing SQL Server assests.☆46Updated 7 years ago
- A curated list of resources about Snowflake☆255Updated last year
- Resources for video demonstrations and blog posts related to DataOps on AWS☆183Updated 4 years ago
- Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.☆168Updated 2 years ago
- This repo contains DAGs demonstrating a variety of ELT patterns using Airflow along with dbt.☆12Updated 3 years ago
- Airflow basics tutorial☆397Updated 4 years ago
- Airflow Unit Tests and Integration Tests☆261Updated 3 years ago
- Data engineering interviews Q&A for data community by data community☆66Updated 5 years ago
- Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks☆162Updated last year
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆28Updated 3 years ago
- Airflow training for the crunch conf☆105Updated 7 years ago
- Snowflake Cookbook, published by Packt☆82Updated 3 years ago
- ☆64Updated 8 months ago
- Code to build a simple analytics data pipeline with Python☆102Updated 8 years ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆141Updated 5 years ago
- ☆16Updated 6 years ago
- Lab Instructions for Data Engineering Immersion Day☆196Updated last week
- Redshift Python Connector. It supports Python Database API Specification v2.0.☆217Updated last week