Understanding the core concepts behind the Lume Python SDK
lume.run()
execution.
lume.run()
.
Each run is defined by two key parameters:
flow_version
: The immutable logic to execute.source_path
: A string that tells Lume what specific data to process. See Understanding source_path
below for details.run.wait()
is great for simple scripts and getting started, we strongly recommend using Webhooks for production applications. They are more scalable and efficient than continuous polling.source_path
source_path
parameter is a string that uniquely identifies the data your pipeline will process. Its meaning depends on the type of Source Connector used by your Flow Version.
source_path
is the full URI to a specific file.
s3://my-customer-data/new_records.csv
source_path
is not a direct path but rather a logical identifier for a batch of data. It’s a string you provide (e.g., a batch ID, a date range) that your pre-configured query in the Lume UI uses to select the correct rows.
"batch_202407291430"
source_path
to filter the data. For example, your query might look like: SELECT * FROM invoices WHERE batch_id = :source_path;
This design prevents SQL injection and separates orchestration logic (the source_path
your code provides) from data access logic (the query managed in Lume).
For a complete, step-by-step guide to running your first pipeline, see the Quickstart Guide.
FAILED
, PARTIAL_FAILED
, or CRASHED
.
Status | Description |
---|---|
CREATED | Run has been accepted and is waiting to be scheduled. |
SYNCING_SOURCE | Lume is actively ingesting data from your source system into its secure staging area. |
TRANSFORMING | The data transformation logic is being executed on the staged data. |
SYNCING_TARGET | Lume is writing the transformed data and metadata to your target system. |
SUCCEEDED | The entire pipeline, including both sync steps and the transformation, completed successfully. |
PARTIAL_FAILED | The pipeline completed. Some rows were transformed successfully, while others were rejected due to validation or mapping errors. Both mapped and rejected data are written to the target system. See Handling Partial Failures for details. |
FAILED | A non-recoverable error occurred during one of the steps. Check metadata for details. |
CRASHED | A fatal system error occurred. Contact support. |