Promoting Data Pipelines to Different Stages in DPS

While working with a data pipeline the Dev stage is the default working stage for you, this means that you can make changes to the pipeline only in the Dev stage. You can however, promote different versions of a data pipeline to various stages for testing. At any given time, each stage can have different versions of the data pipeline depending on which versions are promoted to which stage.

In all other stages apart from the Dev stage, you can switch the data pipeline version to another one using the Available Pipeline Versions option.

In the above image lets take the example of the QA stage with version 1.0.1.6. The pipeline version has 4 bits: Major.Minor.Promotion.Revision.

  • Major - this is incremented when a pipeline is deployed.

  • Minor - this is used for a hotfix release.

  • Promotion - this is incremented when a pipeline is promoted to a succeeding stage.

  • Revision - this is incremented with every published update to the data pipeline or switch to another version of the pipeline.

Consider that we have Dev, QA, and Prod stages.

Note:

The stages that you have depend on the stages added to the Deploy phase of the product. If you are using a project template for your product, the stages are based on the settings in the template.

The table below shows the user actions like changes to the pipeline or promoting the pipeline to a certain stage, and its effect on pipeline versions in the various stages.

User Action Current version in Dev stage Current version in QA stage Current version in Prod stage Comments
You create a data pipeline, make several changes to it, publish the changes. The current version of your pipeline is 1.0.0.8 1.0.0.8 NA NA  
You promote the data pipeline from Dev stage to QA stage. 1.0.1.0 1.0.0.8 NA  

You make changes to the data pipeline and the current version of the data pipeline is now 1.0.1.1.

You promote the pipeline from QA stage to Prod stage.

 1.0.2.0 1.0.1.1 1.0.1.1 When you promote the pipeline from QA to Prod, the promotion is incremented.
You promote the pipeline from Dev stage to QA stage. 1.0.3.0 1.0.2.0 1.0.1.1 When you promote the pipeline from Dev to QA, the promotion is incremented.
You make changes to the pipeline and publish the changes each time. 1.0.3.4 1.0.2.0 1.0.1.1 You make changes to the pipeline 4 times.

 

Related Topics Link IconRecommended Topics What's next? Data Pipeline Stages