Anomaly Detection Results

The Anomaly Detection Results section provides machine learning based insights for analyzer metrics configured with anomaly detection. The system compares current execution results against historical metric behavior and predicted baseline values to identify unusual data quality patterns.

Prerequisites

• Enable Anomaly or Auto-Anomaly and specify Weightage as needed in the Rule Configuration step of the Data Quality Processor screen. For more information, see Data Quality (DQ) Processor using Unity Catalog.

• Generate historical execution data is required before anomaly detection results become available.

Anomaly Summary

After a Data Quality Processor execution completes, the Data Quality Processor Results dashboard displays anomaly information for configured analyzer rules.

The anomaly summary includes:

• Total anomalies detected

• Anomaly Details

Total Anomalies Detected

View the total number of anomalies detected. In order to investigate further, you must click View Anomaly Details. Alternately you can also select an individual metric and view its anomaly details.

View Anomaly Details

Click View Anomaly Details to review detected anomalies.

The Anomalies side drawer opens with the number of detected anomalies.

The detected anomalies table displays the following information:

• Anomaly Observations

• Evaluated Statistics

• Evaluated Value

• Predicted Value

• Severity

• Decision Reason

• Anomaly Status with the following actions:

  • Accept

  • Reject

  • Trend Chart

Accepting Anomalies

To confirm anomaly observations

1. Select one or more Anomaly Observations, then click Accept. A confirmation dialog box appears.

   

   Note:

   The anomaly records are accepted by default and display the status as Accepted.

2. Enter the acceptance reason in the textbox, then click Save and Accept. The Anomaly Status for the selected observations is updated to Accepted and stores the feedback for future model training.

   

   Click the user icon next to the Accepted status of an anomaly observation to view:

   • The name of the user who updated the decision

   • The date on which the anomaly was accepted

   • The time at which the anomaly was accepted

3. To save the modified anomalies and retrain the model, click Train Model. A confirmation dialog appears.

4. Click Train. The accepted anomaly feedback is used to improve future anomaly prediction accuracy.

   

Rejecting Anomalies

To mark observations as non-anomalies

1. Select one or more Anomaly Observations, then click Reject. A confirmation dialog box appears.

2. Enter the rejection reason in the textbox, then click Save and Reject. The Anomaly Status for the selected observations is updated to Rejected and stores the feedback for future model training.

   

   Click the user icon next to the Rejected status of an anomaly observation to view:

   • The name of the user who updated the decision.

   • The date on which the anomaly was rejected.

   • The time at which the anomaly was rejected.

3. To save the modified anomalies and retrain the model, click Train Model. A confirmation dialog appears.

4. Click Train. The rejected anomaly feedback is used to improve future anomaly prediction accuracy.

   

Analyzing Anomaly Trends

You can analyze anomaly trends either from the Anomalies side drawer or from the Deviation Analysis for Analyzer Metrics section.

Analyzing Anomaly Trends via Anomalies

To view trend analysis from Anomalies

• In the Anomaly Status column of the Anomalies side drawer, select the trend chart icon beside each anomaly record to view its additional trend analysis. A trend chart popup opens.

  

  The trend visualization displays:

• Historical run comparison

• Run date and time

• Actual metric value

• Baseline predicted value

• Upper Limit

• Lower Limit

• Severity indicators

Analyzing Anomaly Trends via Deviation Analysis for Analyzer Metrics

To view trend analysis from Deviation Analysis for Analyzer Metrics

1. In the Deviation Analysis for Analyzer Metrics section, click on the required metric with Anomaly Detected icon. A trend chart popup opens with the Anomaly and Last 5 Runs vs. Current Run tabs.

    

   

2. Click on the Anomaly tab. A trend chart displays the metrics.

    

   

   The Anomaly tab includes the following:

   • Historical run comparison

   • Run date and time

   • Actual metric value

   • Baseline predicted value

   • Upper Limit

   • Lower Limit

   • Severity indicators

   • Explanation describing why the metric is classified as anomaly or non-anomaly. The description includes:

     • The analyzed metric

     • Current run value

     • Predicted baseline value

     • Deviation from expected range

     • Severity assessment

   • Anomaly Status with the user icon to view:

     • The name of the user who updated the decision

     • The date on which the anomaly was accepted or rejected

     • The time at which the anomaly was accepted or rejected

   • Decision Reason

     Use the expand or collapse icon in the top-right corner of the trend chart to switch between the expanded and collapsed views of the trend analysis visualization.

3. To accept or reject anomaly, click Accept Anomaly or Reject Anomaly as needed.

   Anomaly detection is available only for analyzer rules configured with Anomaly or Auto-Anomaly enabled.

   The prediction accuracy is continuously improved using the historical run data and review feedback.

   Feedback submitted through Accept, Reject, and Train Model actions contributes to future model training.

Recommended Topics

What's next? Deviation Analysis for Analyzer Metrics