Data Pipeline Chaining

Data Pipeline Studio provides the ability of pipeline chaining. Using this feature, you can automate the run of multiple pipelines based on a specific event of other pipelines. You can schedule pipeline run of one pipeline based on the following events of another pipeline:

On successful completion of a pipeline run
On failure of pipeline run
On timeout of pipeline run
When the pipeline goes into the wait state
On initiation of pipeline run
On completion of pipeline run

Use cases

Let us discuss some use cases where pipeline chaining can be used:

When multiple pipelines need to run in a specific sequence

In this scenario, you may have pipelines A, B, and C. You may want pipeline B to run on completion of run of pipeline A. Then you may want pipeline C to be triggered on successful completion of pipeline B.
When you have pipelines as a fallback for other pipelines

In the scenario, you may have pipelines A, B, C, and D.

You may want to trigger pipeline B on successful run of pipeline A, run pipeline D on timeout of pipeline C, or run pipeline B if pipeline A goes into wait mode.

To create a chained pipeline

Create pipeline 1 and pipeline 2. Let us assume that pipeline 2 has a dependency that the pipeline is triggered on successful run of pipeline 1.
On the home page of Data Pipeline Studio, open pipeline 2 and click the ellipsis (...). Then click Schedule Run.

Click the Event-Based tab and click Add Event to create a schedule with the following options:

Action	Select an action to perform on the dependent pipeline. Choose from the following options: Run Pipeline Stop Pipeline
Event Trigger	The selected action can be triggered by one of the following events: On Completion - on completion of the pipeline, irrespective of whether the pipeline run is successful or not. On Failure - on failure of pipeline run. On Start - on initiation of pipeline run. On Success - on successful completion of pipeline run. On Timeout - if the pipeline run times out. On Wait - if the pipeline goes into wait state.
Product	Select a product from the dropdown list to which the dependent pipeline belongs.
Feature	Select a feature from the dropdown list to which the dependent pipeline belongs.
Stage	Select the stage to which the dependent pipeline belongs.

Click Add to create the event-based schedule.
Run pipeline 1 manually. After pipeline 1 reaches the selected event, the specified action for pipeline 2 is triggered automatically.

Verifying whether pipeline chaining was successful

There are several ways to verify whether pipeline chaining was successful or not.

Consider that pipeline B is dependent on pipeline A. Check the status of the event added for pipeline A. If it is Successful, then the action of pipeline B must be initiated because pipeline A has reached the required status.

For example if the action for pipeline B is run pipeline and event for pipeline A is On Success, then the run for pipeline B must be initiated on successful run of pipeline A.

Check the Audit Logs from the ellipsis (...). In the audit logs you can view the completed pipeline A and then the pipeline gets triggered after the completion of pipeline A.

Note:

The Initiated By column shows System for a pipeline run that is automatically initiated using pipeline chaining.

What's next? Promoting Data Pipelines to Different Stages in DPS