With 12 manufacturing facilities in 5 countries, Micron has valuable business data dispersed around the world. Collecting, collating, validating and presenting this data has proven to be a significant challenge. After hand building dozens of ingestion pipelines, and considering the significant amount of pipelines remaining, it became clear that automation was the only path to success. Rising to the challenge, Micron engineers designed and developed Pipeline Builder, a Java and Angular based platform that automates the deployment of NiFi pipelines to acquire, transmit and ingest data into the Global Data Warehouse. Pipeline Builder has since been used to create and monitor hundreds of ingestion pipelines into the Global Data Warehouse. With so many concurrently active pipelines, it can be difficult to understand the health of each pipeline. Pipeline Manager and Pipeline Validator were created to track each pipeline's health and run automated validation routines to ensure data is transferred accurately from source to target.
Dataworks Summit Washington DC,Dataworks Summit 2019,DWS19,
With 12 manufacturing facilities in 5 countries, Micron has valuable business data dispersed around the world. Collecting, collating, validating and presenting this data has proven to be a significant challenge. After hand building dozens of ingestion pipelines, and considering the significant amount of pipelines remaining, it became clear that automation was the only path to success. Rising to the challenge, Micron engineers designed and developed Pipeline Builder, a Java and Angular based platform that automates the deployment of NiFi pipelines to acquire, transmit and ingest data into the Global Data Warehouse. Pipeline Builder has since been used to create and monitor hundreds of ingestion pipelines into the Global Data Warehouse. With so many concurrently active pipelines, it can be difficult to understand the health of each pipeline. Pipeline Manager and Pipeline Validator were created to track each pipeline's health and run automated validation routines to ensure data is transferred accurately from source to target.
0 Comments