Quickstart

These examples are meant to help you quickly get started with a data mapping pipeline. See the full list of guides for more advanced operations.

​

Running a pipeline

We are ingesting data from an API and want to normalize each record to our own internal schema. The data in question contains information about our users. The API gives us the following source data:

[
    {
        "name": "Barry Devlin",
        "nickname": "Baz",
        "years_old": 42
    },
    {
        "name": "Paul Murphy",
        "nickname": "Polly",
        "years_old": 37
    }
]

Our internal schema can be represented in JSON Schema format as follows:

{
    "type": "object"
    "properties": {
        "first_name": {
            "description": "The first name of the user.",
            "type": "string"
        },
        "last_name": {
            "description": "The last name of the user.",
            "type": "string"
        },
        "age": {
            "description": "The age of the user in years.",
            "type": "integer"
        }
    },
    "required": [
        "first_name",
        "last_name",
        "age",
    ]
}

All we need to do is create a pipeline with our target schema and run it with our source data to retrieve the mappings:

And just like that, Lume completely automates the mapping process! Our mapped data will look like this:

[
    {
        "first_name": "Barry",
        "last_name": "Devlin",
        "age": 42
    },
    {
        "first_name": "Paul",
        "last_name": "Murphy",
        "age": 37
    }
]

​

Workshopping a pipeline

We’ve been successfully running our pipeline on new data, but we’ve received feedback that users prefer being addressed by their nicknames instead of their first names. To accommodate this, we need to update the pipeline so that the first_name field maps from nickname in the source data, rather than name.

To make this change, we’ll create an editing session to edit the pipeline, test the staged changes, and deploy the edits once we are satisfied:

Now any future runs we do with this pipeline will apply the new workshopped logic. For example, rerunning the original source data will give us the following mapped data:

[
    {
        "first_name": "Baz",
        "last_name": "Devlin",
        "age": 42
    },
    {
        "first_name": "Polly",
        "last_name": "Murphy",
        "age": 37
    }
]

​

Next steps

As our data mapping needs grow, we will begin creating more pipelines. Start with Managing Pipelines and read the rest of our guides to learn more!