Supported technologies for connecting nodes in Data Pipeline Studio

While creating a data pipeline, you connect nodes of various stages to each other. The technologies that are supported in the stages depend on the capability you are using such as data integration, data transformation, data quality or data visualization. You must understand how the different stages of a data pipeline can be connected, depending on which technology you have added to the stage.

The following section lists the capabilities with the technology you choose and the supported combination of source and target nodes in Data Pipeline Studio.

This is what a typical data ingestion pipeline looks like in the Lazsa Platform:

Data Integration Pipeline

The Lazsa Platform supports the following technologies for data integration stage in a data ingestion pipeline:

The following technologies are supported for source and target stages using Databricks for data integration:

Source Stage Target Stage
CSV Amazon S3
Snowflake
MS Excel Amazon S3
Snowflake
Parquet Amazon S3
Snowflake
FTP Amazon S3
Snowflake
SFTP Amazon S3
Snowflake
REST API Amazon S3
Snowflake
Lazsa Ingestion Catalog Amazon S3
Snowflake
Microsoft SQL Server Amazon S3
Snowflake
MySQL Amazon S3
Snowflake
Oracle Amazon S3
Snowflake
PostgreSQL Amazon S3
Snowflake
Snowflake Amazon S3
Snowflake
Amazon S3 Amazon S3
Snowflake
Kinesis Streaming Amazon S3
Snowflake
Source Stage Target Stage
Salesforce Amazon S3
Snowflake
ServiceNow Amazon S3
Source Stage Target Stage
Amazon S3 (Data Lake) Snowflake
Source Stage Target Stage
Amazon S3 (Data Lake) Snowflake

This is what a typical data transformation pipeline looks like in the Lazsa Platform:

Data Transformation Pipeline

The Lazsa Platform provides data transformation using templatized jobs or custom jobs depending on the code that you use. Templatized jobs include join/union/aggregate functions that can be performed to group or combine data. For complex operations to be performed on data, DPS provides the option of creating custom transformation jobs.

The Lazsa Platform supports data transformation using the following technologies:

Data Lake Target Stage
Snowflake (custom transformation) Snowflake
Amazon S3 Amazon S3
Source Stage Target Stage
Snowflake Snowflake

This is what a typical data quality pipeline looks like in the Lazsa Platform:

You can perform data quality using Databricks or Snowflake data quality capabilities, on the dataset available in an Amazon S3 data lake or a Snowflake data lake. The technology that you use entirely depends on the organizational preference.

The Lazsa Platform provides various tools for data quality:

Data Quality Stage Target Stage
Data Profiler Amazon S3
Data Analyzer Amazon S3
Data Issue Resolver Amazon S3
Data Quality Stage Target Stage
Data Profiler Snowflake
Data analyzer Snowflake
Data Issue Resolver Snowflake

This is what a typical data visualization pipeline looks like in the Lazsa Platform.

Data Visualization pipeline

The following technologies are supported for data visualization:

Data Lake
Snowflake

Analytical Data Store - the following RDBMS technologies are supported:

  • Amazon RDS for MySQL

  • Amazon RDS for PostgreSQL

  • Microsoft SQL Server

  • MySQL

  • Oracle

  • PostgreSQL

  • Snowflake

This is what a typical data analytics pipeline looks like in the Lazsa Platform.

Data Analytics pipeline

The Lazsa Platform provides the option of using either predefined algorithms like Random Forest Classifiers, Support Vector Classifier or creating custom algorithms according to your specific requirement.

Source Stage Target Stage
Snowflake Snowflake
Amazon S3 Snowflake
Snowflake Amazon S3
Amazon S3 Amazon S3

 

Related Topics Link IconRecommended Topics

What's next?Create a Data Pipeline