GoogleCloudPlatform / Data-Pipeline
Data pipeline is a tool to run Data loading pipelines. It is an open sourced app engine app that users can extend to suit their own needs. Out of the box it will load files from a source, transform them and then output them (output might be writing to a file or loading them into a data analysis tool). It is designed to be modular and support var…
☆88Updated 10 years ago
Related projects ⓘ
Alternatives and complementary repositories for Data-Pipeline
- Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.☆163Updated 7 years ago
- *luigi-gcloud* is an luigi extension that enables full support for the Google Cloud Platform. Making it possible to do complex orchestrat…☆42Updated 8 years ago
- Example Kubernetes app that shows how to build a 'pipeline' to stream data into BigQuery. Uses Redis or Google Cloud PubSub☆129Updated 4 years ago
- ☆54Updated 7 years ago
- Processing Logs at Scale using Cloud Dataflow☆61Updated 5 years ago
- Cloud Pub/Sub sample applications with Python☆72Updated 8 years ago
- ☆84Updated 6 years ago
- This is the support code and solutions for the NYC Taxi Tycoon Dataflow Codelab☆60Updated 5 years ago
- Opinion Analysis of News, Threaded Conversations, and User Generated Content☆101Updated 2 months ago
- Repository with examples and smoke tests for the GCP Airflow operators and hooks☆146Updated 7 years ago
- An example that shows how to periodically launch a Dataflow analytics pipeline from GAE Flex, that reads from Datastore.☆42Updated 7 years ago
- makeViewerUrl☆85Updated 6 months ago
- Simplest way to get Tweets into BigQuery. Uses Google Cloud & App Engine, as well as Python and D3.☆141Updated 8 years ago
- Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines. This re…☆166Updated 6 years ago
- Example stream processing job, written in Scala with Apache Beam, for Google Cloud Dataflow☆30Updated 7 years ago
- Example that shows use of the Prediction API☆10Updated 6 years ago
- Sample app exercising gcloud-python library☆67Updated 6 years ago
- ☆64Updated 3 months ago
- ☆46Updated 6 months ago
- Google Datalab Library☆194Updated 2 years ago
- Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP☆89Updated 3 months ago
- This service is meant to simplify running Google Cloud operations, especially BigQuery tasks. This means you do not have to worry about …☆46Updated 5 years ago
- Demonstrating the concept of Google PubSub, a messaging queue service in Google, thru streaming fake financial data thru PubSub and query…☆15Updated 7 years ago
- Cloud Dataproc: Samples and Utils☆198Updated last week
- Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow☆58Updated 4 years ago
- Spark pipelines that correspond to a series of Dataflow examples.☆27Updated 5 years ago
- Small python logging handlers that directly send the logs to Cloud Pub/Sub☆23Updated 4 years ago
- View billing export files via an App Engine application dashboard.☆20Updated 7 years ago
- A tool for moving tables from Redshift to BigQuery☆65Updated 5 years ago