More Options or Features of Data Pipeline Studio

You can receive notifications for actions like failed or successful data pipeline runs, changes in data pipeline, aborting a data pipeline run, promoting a data pipeline to next stage and so on.

You can configure the following types of notifications:

• Email

• SNS - subscribe to SNS topics to receive notification

• SQS - poll the queue for events. See Trigger a pipeline based on SQS notifications.

For SNS and SQS you must configure the tools from Systems Integration under Cloud Platform Tools and Technologies. See Configure Amazon SNS and Configure Amazon SQS for detailed instructions.

Manage Wheel Package lets you upgrade to a latest available version of wheel package Wheel package is a standard way to package and distribute the files required to run a Databricks job. or change to any available wheel package version.

You can set the time after which a pipeline that is running is timed out. Click the option and provide the following details:

• Default rerun attempts for all nodes of the pipeline - Specify the maximum number of times the node is rerun in case of failure.

• Pipeline Run Timeout - Specify the duration after which the pipeline run is timed out, irrespective of the run status. In case this parameter is not specified, then default setting of 24 hours is considered.



Note:

Depending on the start time of the pipeline run, the pipeline run timeout limit may exceed by a maximum of 5 minutes, which is the threshold limit of the pipeline scheduler.