> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lume.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Core Concepts

<Note>
  **Welcome to Lume!** 🚀 We've recently introduced **Projects** as our new way to organize data transformations. If you're new to Lume or evaluating our platform, we recommend starting with our [Project Guides](/documentation/project_guides/core_concepts) for the latest and greatest experience. Existing customers can continue using Flows while we help you transition.
</Note>

<Note>
  Understanding these foundational concepts will help you make the most of Lume's capabilities. This guide introduces the key components and how they work together.
</Note>

## Data Pipeline Basics

<CardGroup>
  <Card title="Source Data" icon="database" href="#source-data">
    The input data you want to transform
  </Card>

  <Card title="Target Schema" icon="bullseye" href="#target-schema">
    The desired structure for your output data
  </Card>
</CardGroup>

### Source Data

Source data is any user-provided data that you want to interpret or transform. Lume supports various structured and semi-structured formats:

* JSON
* CSV
* Excel
* And more

<Accordion title="Example Source Data">
  ```json theme={null}
  [
      {
          "id": "0018y000008hFqqAAE",
          "name": "Blue Sky Ventures LLC",
          // ...
      },
      {
          "id": "0018y000008nrMrAAI",
          "name": "Green Acres Holdings LLC",
          // ...
      }
  ]
  ```
</Accordion>

<Info>
  While Lume only requires a single record to generate mapping logic, providing larger data samples improves mapping accuracy through better pattern recognition.
</Info>

<Note>
  Need support for additional data formats? Contact the Lume team for assistance!
</Note>

### Target Schema

A target schema defines the desired output format for your transformed data. It uses [JSON Schema](https://www.mongodb.com/basics/json-schema-examples) format to specify:

* Expected data types
* Field requirements
* Data validation rules
* Format specifications

<Note>
  Remember: Property names in Lume's API cannot contain periods (.).
</Note>

<Info>
  Don't know JSON Schema? Lume can automatically generate a target schema from a sample CSV file containing your desired output format. This makes it easy for non-technical users to define their data requirements.
</Info>

<Accordion title="Example Target Schema">
  ```json theme={null}
  {
    "type": "object",
    "properties": {
      "full_name": {
        "type": ["string"],
        "description": "The full name of the customer, including first name and last name."
      },
      "email_address": {
        "type": ["string"],
        "description": "The customer's primary email address used for communication.",
        "format": "email"
      }
      // ... additional fields ...
    },
    "required": ["full_name", "email_address"]
  }
  ```
</Accordion>

<AccordionGroup>
  <Accordion title="Required Schema Properties">
    **Type:** Defines allowed data types

    ```json theme={null}
    {
      "type": ["string", "null"]
    }
    ```

    **Description:** Explains the field's purpose

    ```json theme={null}
    {
      "description": "The status of the business's registration"
    }
    ```
  </Accordion>

  <Accordion title="Optional Schema Properties">
    **Enum:** Defines allowed values for classification

    ```json theme={null}
    {
      "enum": ["CA", "NY", "TX"]
    }
    ```

    **Format:** Specifies data format (email, date, UUID, etc.)

    ```json theme={null}
    {
      "format": "date-time"
    }
    ```

    **Pattern:** Enforces format via regex

    ```json theme={null}
    {
      "pattern": "^\\+?[1-9]\\d{1,14}$"
    }
    ```
  </Accordion>

  <Accordion title="Advanced Properties">
    **cleaning-instructions:** Custom data cleaning rules

    ```json theme={null}
    {
      "lume-settings": {
        "cleaning-instructions": "Remove special characters and convert to lowercase"
      }
    }
    ```

    **source-enum:** Defines acceptable source values for classification

    ```json theme={null}
    {
      "lume-settings": {
        "source-enum": ["California", "New York", "Texas"]
      }
    }
    ```
  </Accordion>
</AccordionGroup>

## Pipeline Components

<CardGroup>
  <Card title="Flows" icon="diagram-project" href="#flows">
    Orchestrate your data transformation journey
  </Card>

  <Card title="Mapping" icon="wand-magic-sparkles" href="#mapping">
    AI-powered data transformation
  </Card>

  <Card title="Runs" icon="play" href="#runs">
    Execute and monitor your transformations
  </Card>
</CardGroup>

### Flows

A flow is your complete data transformation pipeline. It can:

* Accept multiple data inputs
* Include multiple transformation steps
* Join and combine data
* Produce final mapped output

Flows help you organize related transformations into logical sequences. Complex transformations can be broken down into manageable steps, making them easier to maintain and modify.

### Mapping

Lume generates Python code to transform your data, but you don't need to be a programmer to use it effectively. The platform provides:

**For Excel Users:**

* Visual data lineage showing how fields map between source and target
* Sample data previews at each transformation step
* Natural language explanations of the transformation logic
* Interactive workshopping interface for adjusting mappings

**For Developers:**

* Python-based transformation code (with SQL, dbt, and JSONata support coming soon)
* Version control for all mapping logic
* Direct code editing capabilities
* Test-driven development support

<Info>
  Whether you're Excel-proficient or a seasoned developer, Lume provides the tools you need to understand and control your data transformations.
</Info>

<Info>
  Access the Schema Transform node within your flow to workshop and improve your mapper logic. Test changes with sample data before deployment.
</Info>

### Runs

A run represents the actual execution of your flow. Each run tracks:

* Input source data
* Output mapped data
* Mapper version used
* Real-time execution status
* Validation results

## Quality Control

<CardGroup>
  <Card title="Validation" icon="check-double" href="#validation">
    Ensure data quality and accuracy
  </Card>

  <Card title="Monitoring" icon="chart-line" href="#monitoring">
    Track performance and catch issues
  </Card>

  <Card title="Iteration" icon="rotate" href="#iteration">
    Improve and refine your pipelines
  </Card>
</CardGroup>

### Validation

Lume provides comprehensive validation capabilities:

* Schema compliance checking
* Data format verification
* Required field validation
* Custom validation rules

### Monitoring

Monitor your data transformations through:

* Real-time run status
* Field-level validation results
* Macro statistics
* Performance metrics

### Iteration

Improve your pipelines through:

* Direct code inspection
* Interactive workshopping
* Test-driven development
* Version control