Data Analytics using Python with JupyterLab

1. On the home page of DPS, add the following stages and connect them as shown below:
   • Data Lake: Snowflake
   • Data Analytics: Python with JupyterLab
   • Data Lake: Amazon S3

   

2. Configure the Amazon S3 and Snowflake nodes. Ensure that JupyterLab has access to the S3 node in the data pipeline. See Setting up S3 access from JupyterLab.

3. Click the Python with JupyterLab node. Click +Algorithm.

   

4. Provide the following information:

   1. Algorithm Type: Select a Predefined Algorithm or Custom Algorithm to be used for data analytics.

   2. Algorithm Name: Provide a unique name for the algorithm.

   3. Source: Select a table in the source, on which the algorithm will be run.

   4. Algorithm Label: Select a column from the pre-populated list of the source table. This is the column on which the selected algorithm is run.

      

      Note:

      Ensure that the selected column contains binary data only.

   5. Target: Select a table in which the data analytics results will be pushed. You can either select an existing table or create a new table by adding the name <xyz> and then clicking Create <xyz>.

   6. Click Add.

5. After you click Add, the Creation Status displays the following types of status depending on whether the data analytics job for the selected algorithm is created or not:

   1. In Progress - when the job is being created.

   2. Success - when the job is created successfully.

   3. Failed - when the job creation fails.

6. To add multiple algorithms, repeat steps 3-5.

7. To run the algorithms, choose one of the following options:

   1. Publish the pipeline. Click Run to run all the algorithms added to the pipeline.

   2. Click Run to run the individual algorithm.

   3. Click Run All to run the complete list of algorithms.

8. The Run Status of the pipeline can be one of the following:

   1. Running

   2. Success

   3. Failed

   4. Terminated

   5. Aborted

9. Click to view the dashboard. View the dashboard generated from the pickle file.

In case the algorithm fails and you need to troubleshoot it, then click the specific algorithm. Before you do that, click the ellipsis (…) adjacent to +Algorithm and click Copy JupyterLab Token. When you click a specific algorithm, it navigates you to JupyterLab Notebook, you need the token to log in. Once you have logged in, you can make troubleshoot the code.

You can check the logs for the data analytics jobs that are created in Data Pipeline Studio. For each data analytics job, a Jenkins job is triggered. You can navigate to Jenkins from the Lazsa Platform to view the logs.

1. Click the data analytics node. On the side drawer, click CI/CD Pipeline.

   

2. Click Deploy.

   

3. On the Jenkins screen click Console Output in the left navigation pane.

   

4. Search for Lazsa-Core-Jobs run-ml-algorithms. You can view the logs for a specific job.

   

   There are multiple ways to navigate to Jenkins logs from the Lazsa Platform.

   Option	Description
   By clicking the Creation Status of an algorithm.
   By clicking the Run Status of an algorithm.
   By clicking View Details > Select the pipeline run > Click View Logs
   When you click any of the above links, you are navigated to the specific build in Jenkins
   Click Console Output in the left navigation pane to view the logs for the specific build.

If your JupyterLab algorithm shows the run status as Failed, do the following:

1. Click the Run Status. This navigates you to the Jenkins logs for the specific job. You may see the following error:

   

2. In the Lazsa platform, on the home page of Data Pipeline Studio, click the pencil icon to switch the pipeline into edit mode.

3. Click the Python with JupyterLab node.

4. On the Python with JupyterLab side drawer, click Start Coding. You are navigated to JupyterLab Notebook.

   To install Papermill in JupyterLab Notebook terminal

   1. Copy the JupyterLab token from the section above. Use the token and log in to your JupyterLab application.

   2. On the File menu, click New > Terminal. This launches a JupyterLab terminal.

   3. Run the following command in the terminal:

      Copy

      pip install papermill

   4. Run the following command to check the papermill version:

      Copy

      papermill-version

   5. Once the papermill package is installed successfully, run the data pipeline.