Core Concepts

​

Data Pipeline Basics

​

Source Data

Source data is any user-provided data that you want to interpret or transform. Lume supports various structured and semi-structured formats:

• JSON
• CSV
• Excel
• And more

[
    {
        "id": "0018y000008hFqqAAE",
        "name": "Blue Sky Ventures LLC",
        // ...
    },
    {
        "id": "0018y000008nrMrAAI",
        "name": "Green Acres Holdings LLC",
        // ...
    }
]

​

Target Schema

A target schema defines the desired output format for your transformed data. It uses JSON Schema format to specify:

• Expected data types
• Field requirements
• Data validation rules
• Format specifications

{
  "type": "object",
  "properties": {
    "full_name": {
      "type": ["string"],
      "description": "The full name of the customer, including first name and last name."
    },
    "email_address": {
      "type": ["string"],
      "description": "The customer's primary email address used for communication.",
      "format": "email"
    }
    // ... additional fields ...
  },
  "required": ["full_name", "email_address"]
}

{
  "type": ["string", "null"]
}

{
  "description": "The status of the business's registration"
}

{
  "enum": ["CA", "NY", "TX"]
}

{
  "format": "date-time"
}

{
  "pattern": "^\\+?[1-9]\\d{1,14}$"
}

{
  "lume-settings": {
    "cleaning-instructions": "Remove special characters and convert to lowercase"
  }
}

{
  "lume-settings": {
    "source-enum": ["California", "New York", "Texas"]
  }
}

​

Pipeline Components

​

Flows

A flow is your complete data transformation pipeline. It can:

• Accept multiple data inputs
• Include multiple transformation steps
• Join and combine data
• Produce final mapped output

Flows help you organize related transformations into logical sequences. Complex transformations can be broken down into manageable steps, making them easier to maintain and modify.

​

Mapping

Lume generates Python code to transform your data, but you don’t need to be a programmer to use it effectively. The platform provides:

For Excel Users:

• Visual data lineage showing how fields map between source and target
• Sample data previews at each transformation step
• Natural language explanations of the transformation logic
• Interactive workshopping interface for adjusting mappings

For Developers:

• Python-based transformation code (with SQL, dbt, and JSONata support coming soon)
• Version control for all mapping logic
• Direct code editing capabilities
• Test-driven development support

​

Runs

A run represents the actual execution of your flow. Each run tracks:

• Input source data
• Output mapped data
• Mapper version used
• Real-time execution status
• Validation results

​

Quality Control

​

Validation

Lume provides comprehensive validation capabilities:

• Schema compliance checking
• Data format verification
• Required field validation
• Custom validation rules

​

Monitoring

Monitor your data transformations through:

• Real-time run status
• Field-level validation results
• Macro statistics
• Performance metrics

​

Iteration

Improve your pipelines through:

• Direct code inspection
• Interactive workshopping
• Test-driven development
• Version control