Configure Amazon Kinesis Data Streams

Amazon Kinesis Data Streams is a fully-managed, serverless data streaming service that stores and ingests streaming data in real time and at a large scale. Since data is processed as it arrives, it enables you to get timely insights and respond to information quickly. You can process real-time data such as audio, video, application logs, IoT telemetry data to name a few.

After you save your Amazon Kinesis Stream connection details in the Lazsa Platform, you can start using it as a data source in your data pipelines.

The Lazsa Platform offers various options for retrieving the credentials of the data source to establish a secure connection. You can either directly provide the credentials within the connection details, where they are securely stored in the Lazsa-managed secret manager. Alternatively, you can choose to retrieve credentials programmatically from your designated secrets management tool.

To configure the connection details of your Amazon Kinesis Data Streams, do the following:

Sign in to the Lazsa Platform and click Configuration in the left navigation pane.
On the Platform Setup screen, on the Cloud Platform, Tools & Technologies tile, click Configure.
On the Cloud Platform, Tools & Technologies screen, in the Databases and Data Warehouses section, click Configure.
(After you save your first connection details in this section, you see the Modify button here.)

Databases and Data Warehouses Configuration in Lazsa Platform

On the Kinesis Streaming screen, do the following:

In the Details section, provide the following details:

Field	Description
Name	Give a unique name to your Amazon Kinesis Data Streams configuration. This name is used to save and identify your specific Amazon Kinesis Stream connection details within the Lazsa Platform.
Description	Provide a brief description that helps you identify the purpose or context of this Amazon Kinesis Data Streams configuration.

In the Configuration section, provide the following information:

Field	Description
Kinesis URL	Provide the URL for Amazon Kinesis Data Streams.
Region	Specify the region for your AWS account that is associated with the Amazon Kinesis Data Streams.
Stream Name	Provide the name of the stream that is created in AWS, which will be used for data streaming.

Depending on how you want to retrieve the credentials to connect to your Amazon Kinesis Data Streams connection, do one of the following:

Field	Description
Connect using Lazsa Orchestrator Agent	Enable this option to resolve your Amazon Kinesis Data Streams credentials within your private network via Lazsa Orchestrator Agent without sharing them with the Lazsa Platform. Provide the following information: Master AWS Account - Provide the mater account ID for your organization. External ID Key - Provide the unique identifier generated by Calibo. You need to mention this ID in the IAM role policy that you create to allow the Lazsa Platform to access your AWS account. If you use the Calibo-provided CFT for IAM role policy, this ID is already mentioned in the template. Cross Account Role ARN Key - After you create an IAM role and attach a policy to establish a trusted relationship between your AWS account and Calibo's AWS account, you can provide the role's Amazon Resource Name (ARN) here. This ARN is required for the Lazsa Platform to assume the role that you create in your AWS account.
Select Secret Manager	Select Lazsa and type your Amazon Kinesis Data Streams username and password. In this case, the user credentials are securely stored in the Lazsa-managed secrets store. Select AWS Secrets Manager. In the Secret Management dropdown list, the AWS Secrets Manager configurations that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen are listed for selection. Select the configuration of your choice. Provide the Secret Name, for the Lazsa Platform to retrieve the secrets for Amazon Kinesis Data Streams. Select Azure Key Vault. In the Secret Management dropdown list, the Azure Key Vault configurations that you save and activate in the Secret Managementsection on the Cloud Platform, Tools & Technologies screen are listed for selection. Select the configuration of your choice. Provide the Vault Name, Username Secret, and Password Secret for the Lazsa Platform to retrieve the credential values.

Field

Description

Connect using Lazsa Orchestrator Agent

Enable this option to resolve your Amazon Kinesis Data Streams credentials within your private network via Lazsa Orchestrator Agent without sharing them with the Lazsa Platform.

Provide the following information:

Master AWS Account - Provide the mater account ID for your organization.
External ID Key - Provide the unique identifier generated by Calibo.
You need to mention this ID in the IAM role policy that you create to allow the Lazsa Platform to access your AWS account. If you use the Calibo-provided CFT for IAM role policy, this ID is already mentioned in the template.
Cross Account Role ARN Key - After you create an IAM role and attach a policy to establish a trusted relationship between your AWS account and Calibo's AWS account, you can provide the role's Amazon Resource Name (ARN) here. This ARN is required for the Lazsa Platform to assume the role that you create in your AWS account.

Kinesis Stream Orchestrator Agent

Select Secret Manager

Select Lazsa and type your Amazon Kinesis Data Streams username and password.

In this case, the user credentials are securely stored in the Lazsa-managed secrets store.
Select AWS Secrets Manager. In the Secret Management dropdown list, the AWS Secrets Manager configurations that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen are listed for selection. Select the configuration of your choice. Provide the Secret Name, for the Lazsa Platform to retrieve the secrets for Amazon Kinesis Data Streams.
Select Azure Key Vault. In the Secret Management dropdown list, the Azure Key Vault configurations that you save and activate in the Secret Managementsection on the Cloud Platform, Tools & Technologies screen are listed for selection. Select the configuration of your choice. Provide the Vault Name, Username Secret, and Password Secret for the Lazsa Platform to retrieve the credential values.

Provide the following information related to the AWS account that is associated with Amazon Kinesis Data Streams:

Field Description

Master AWS Account Provide your organization's AWS master account ID.

External ID Key This is the unique identifier generated by Calibo. You need to mention this ID in the IAM role policy that you create to allow the Lazsa Platform to access your AWS account. If you use the Calibo-provided CFT for IAM role policy, this ID is already mentioned in the template. You can copy this ID for your reference.

Cross Account Role ARN Key After you create an IAM role and attach a policy to establish a trusted relationship between your AWS account and Calibo's account, you can provide the role's Amazon Resource Name (ARN) here. This ARN is required for the Lazsa Platform to assume the role that you create in your AWS account.

Add more Kinesis configuration parameters
You can add additional parameters to the Kinesis configuration. These parameters help you to narrow down the reading of the data stream as required.

Select a parameter from the following options:

shardId - The ID of the Kinesis Data Streams shard to get the iterator for.

limit -

sharditeratorType - This determines how the shard iterator is used to start reading data records from the shard.

timestamp - This is the time stamp of the data record from which to start reading.

startingSequenceNumber - This is the sequence number of the data record in the shard from which to start reading.

Provide a value for the parameter.

Click Add.

Click Test Connection to validate whether you have configured the correct connection details and you can connect to Amazon Kinesis Data Streams successfully.

Secure configuration details with a password
To password-protect your Amazon Kinesis Data Streams connection details, turn on this toggle, enter a password, and then retype it to confirm. This is optional but recommended. When you share the connection details with multiple users, password protection helps you ensure authorized access to the connection details.

Click Save Configuration. The configured connection details, you can see the configuration listed on the Databases and Data Warehouses screen.

Field	Description
Master AWS Account	Provide your organization's AWS master account ID.
External ID Key	This is the unique identifier generated by Calibo. You need to mention this ID in the IAM role policy that you create to allow the Lazsa Platform to access your AWS account. If you use the Calibo-provided CFT for IAM role policy, this ID is already mentioned in the template. You can copy this ID for your reference.
Cross Account Role ARN Key	After you create an IAM role and attach a policy to establish a trusted relationship between your AWS account and Calibo's account, you can provide the role's Amazon Resource Name (ARN) here. This ARN is required for the Lazsa Platform to assume the role that you create in your AWS account.
Add more Kinesis configuration parameters	You can add additional parameters to the Kinesis configuration. These parameters help you to narrow down the reading of the data stream as required. Select a parameter from the following options: shardId - The ID of the Kinesis Data Streams shard to get the iterator for. limit - sharditeratorType - This determines how the shard iterator is used to start reading data records from the shard. timestamp - This is the time stamp of the data record from which to start reading. startingSequenceNumber - This is the sequence number of the data record in the shard from which to start reading. Provide a value for the parameter. Click Add.

What's next? Configure Technologies and Testing Tools