Managing Pipelines
How to organize, manage, and leverage existing pipelines
Learn more about Pipelines here.
Managing Pipelines with Lume
With Lume, you can build and manage multiple pipelines, potentially even hundreds or thousands. As an API, Lume can be embedded anywhere in your code, fitting seamlessly into your pipeline management architecture.
Key Aspect: Pipeline Names
Pipeline names are crucial for tracking and reusing deployed pipelines. Use these names to route specific data to the appropriate pipelines.
Example:
If you are normalizing data from multiple data feeds (Feed A, Feed B, and Feed C) into one internal schema, you could create three pipelines with the following names:
FeedA_to_internalFeedB_to_internalFeedC_to_internal
When data comes in from Feed A, route it to the FeedA_to_internal pipeline.
Refer to Use Pipeline to learn more.
Best Practices for Naming Pipelines
- Organize Your Pipelines: Use pipeline names to organize your pipelines. For example, create different pipelines per customer, per integration, or per customer system.
- Code-Retrievable Names: Ensure names are retrievable via code. Sometimes, using a hash can work well if you want to embed the pipelines.
Leveraging Existing Pipelines
- Reuse for Consistency: Leverage existing pipelines if you want to use the same mapper on new data to guarantee consistent output.
- Create New Pipelines for New Mappers: Generate new pipelines if you need to create new mappers for different data sets.
For more detailed guidance, refer to our Getting Started documentation.