Pipeline Version and Wheel Package Version

This section helps you understand wheel packages, their versions, and how they are related to pipeline versions.

What is a wheel package?

A wheel package is a packaged Python library that contains reusable code required to run a pipeline. It is distributed as a .whl file and can include Python modules, functions, and their associated dependencies.

What is the effect of switching pipeline versions on wheel package versions?

Wheel packages are required to run jobs. When you create a data pipeline using Databricks or Snowpark switching to a different version of the data pipeline might also change the wheel package version. Click View Details to understand the impact of switching the pipeline version on wheel package version.

A blinking dot above the ellipsis (…) on the landing page of DPS indicates that a wheel package upgrade is available. You can click the ellipsis and then click Manage Wheel Package.

Wheel pakage upgrade indicator

The Manage Wheel Package side drawer provides the following options:

Field Description
Update to Latest Allows you to update the wheel package to the latest available version.
Change Displays the available versions of the wheel package and lets you select a version to which you want to switch.
Wheel Package Status

This section displays the information about the wheel package version and the instances that currently have that wheel package version installed. This information is displayed in a tabular format:

Version Instances Involved
Version XYZ Databricks123, Databricks456
Version ABC Snowpark789, Snowpark01

 

Related Topics Link IconRecommended Topics What's next? Promoting Data Pipelines to Different Stages in DPS