Handling Pipeline Failures in Data Pipeline Studio

Complex pipelines take a lot of effort and time to create and run. Each such pipeline can consist of a number of jobs depending on the various stages it comprises of. Pipeline run failures of such pipelines can incur a considerable cost to an organization. It thus becomes very critical to handle pipeline failures optimally.

Data Pipeline Studio handles pipeline failures in such a way that resources are utilized optimally, yet ensuring that the runs are successful with desired results, and keeping an overall control on the costs. You can use features like number of retry attempts, pipeline run timeout, and resume pipeline run to ensure that a pipeline does not go into a hung state. If the pipeline run fails at a certain stage, you can resume the run from the failed node. In this case, the preceding jobs that were successful do not have to be run again. This is applicable to all types of jobs – integration, transformation, data quality and so on.

The parameters that help you to handle pipeline failures are as follows:

Pipeline Timeout

Pipeline Run Timeout terminates pipeline runs automatically, beyond the specified duration. This ensures that the pipeline does not go into a hung state in case of failure.
Default Rerun Attempts for all nodes of a pipeline is the number of times the pipeline run is reinitiated upon the first pipeline run failure.
Node Rerun Attempts is the number of times a specific node of the pipeline runs upon the first failure.
Resume Pipeline is resuming the pipeline run from a point of failure.

How do I set the parameters for handling pipeline failures?

After you publish a pipeline, on the ellipsis (…) click Pipeline Run Timeout.
Set the Default rerun attempts for all nodes of a pipeline. You can change the default setting and set Node Rerun Attempts individually, at the node level.
Set the Pipeline Run Timeout in hours and minutes.
Click a node of a job in the pipeline for which you want to change the node-level setting and set the Node Rerun Attempts for it.

Note:

The setting Node Rerun Attempts done at the node level overrides the setting Default rerun attempts for all nodes of a pipeline which is done at the pipeline level.