Configure Connection Details of DevOps CI/CD Pipeline Tool
DevOps CI/CD pipeline tools are software tools and platforms that facilitate the implementation and management of Continuous Integration (CI) and Continuous Deployment (CD) practices within DevOps workflows. These tools are designed to automate and streamline the process of building, testing, and deploying software applications, enabling teams to deliver high-quality software more efficiently and reliably. They offer a range of features and functionalities such as source code management, build automation, testing and quality assurance, deployment automation, continuous monitoring, and orchestration and workflow management, among others.
Jenkins
Currently, the Lazsa Platform supports Jenkins in the DevOps CI/CD Pipeline Configuration category. Before you configure the connection details of your Jenkins account in the Lazsa Platform, ensure that you have completed the following prerequisites:
Prerequisites and Required Permissions
-
The Jenkins user must have administrative privileges.
-
The Jenkins user must have a permission to copy SSH keys to the Jenkins server and agents.
Required Permissions
The following table contains the details of the user inputs required in the Lazsa Platform and the required permissions to process the inputs.
| User Input required in Lazsa | Required Permissions | Additional Details |
|---|---|---|
|
Write |
Job
Credentials
Jenkins
Agent
View
SCM
Matrix
User Permissions (If Matrix authentication is enabled)
Required Software / Plugins / Slave Tokens Software
Plugins
Slave
|
Custom AMI for Docker Deployment
If you plan to use a Docker mode for deploying technologies in the Lazsa Platform, create an AMI by using the following steps. This AMI needs to be available in the AWS cloud account/region where you plan to deploy the technologies. You need to provide an AMI ID in the AWS account configuration of the Lazsa Platform. The steps mentioned in this section are for creating an AMI on Amazon Linux 2 AMI 2.0.20210126.0 x86_64 HVM gp2. If you use any other OS, use the equivalent commands. Refer to the following steps to install and verify all the required packages. Execute the given commands in the same sequence. You need root user privileges or a user with sudo permissions to execute these commands:
-
Sign in to the AWS account that you want to configure in the Lazsa Platform.
-
Create an EC2 Instance with the OS Amazon Linux 2 AMI 2.0.20210126.0 x86_64 HVM gp2.
-
Execute the following commands on the EC2 instance by using the “root” user or use “sudo”.
It should be Amazon Linux 2 AMI 2.0.20210126.0 x86_64 HVM gp2. If you are using any other OS, use the equivalent commands.
-
Check OS version on your machine.
It should be Amazon Linux 2 AMI 2.0.20210126.0 x86_64 HVM gp2. If you are using any other OS, use the equivalentcommands.
Copycat /etc/os-release -
Update the packages on your machine.
Copyyum update -yUpdate the packages on your system.
- Install the required packages.
Copy
i. yum -y install java-1.8.0-openjdk
ii. yum -y install git
This is required to clone Git repositories on the machine on which you deploy the technology stack.
Copyiii. yum -y install jqThis is required to slice, filter, map, and transform structured data.
Copyiv. yum -y install gcc
v. yum -y install unzipThis is required to extract .zip files.
Copyvi. yum -y install gcc-c++
vii. yum -y install python3This is required for technologies like Databricks, Jupyterlab3.0.7, etc.
Copyviii. y um -y install python3-pipThis is required for technologies like Jupyterhub, Jupyterlab3.0.7, etc.
-
Install Docker.
Copyamazon-linux-extras install docker
systemctl enable docker
systemctl start dockerInstall Portainer Server container –
Copydocker run -d -p 8000:8000 -p 9000:9000 --name=portainer --
restart=always -v /var/run/docker.sock:/var/run/docker.sock -v
portainer_data:/data portainer/portainer-ce --admin-password '<Use
hash generated using the following Docker command>'
Portainer provides a Docker UI that helps you visualize containers, images, volumes, and networks.
You can generate new password for Portainer by using the following command:
Copydocker run --rm httpd:2.4-alpine htpasswd -nbB admin '<your-secret-
password>' | cut -d ":" -f 2
- Create a “ciuser” User.
This Linux user is used by Jenkins to connect to machines on which you deploy the tech stack.Copyuseradd -d "/home/ciuser" ciuser - Set a password for the “ciuser” userCopy
echo '<password>' | sudo passwd --stdin ciuserGet the password from your System Admin/DevOps team, replace <password> with the password for the “ciuser” Linux User.
- Add the “ciuser“ User to the Docker group.
Enable Key-Based authentication.
To enable Key-Based authentication between Jenkins (Master/Slave) and the machines on which you deploy the tech stack created using the AMI, your Jenkins Server must have a private key added to it and its corresponding public key should be added in the authorized_keys file of this AMI.
To do so, perform the following steps.mkdir /home/ciuser/.ssh
The authorized_keys files must be in the /home/ciuser/.sshtouch /home/ciuser/.ssh/authorized_keys
Create an empty /home/ciuser/.ssh/authorized_keys.Add the public key to
/home/ciuser/.ssh/authorized_keys
Use a text editor to add the key. For example, vim /home/ciuser/.ssh/authorized_keys
Change the owner of /home/ciuser directory.
Copychown -R ciuser:ciuser /home/ciuser- Set permissions.Copy
chmod 700 /home/ciuser/.ssh
chmod 600 /home/ciuser/.ssh/authorized_keys
Set these permissions so that no other Linux user except the owner, i.e. “ciuser“, can read/write/execute /home/ciuser/.ssh/authorized_keys
Provide sudo permissions to the “ciuser” user. Give the “ciuser” sudo permission. You may need suitable permissions to run this command -(The ciuser needs sudo permission as packages need to be installed for data tech stack.)
Copyecho 'ciuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers- Restart the sshd service, if required.Copy
service sshd restart - Execute the script that validates dependencies. Run the following script: Copy
#!/bin/bash
printf "This script has been developed and tested for Amazon Linux OS. If you are using any other OS or facing any issues, please get back to us.\n\n"
## Unset a bunch of variables we will use later
unset machinejavaversion predefinedjavaversion machinegitversion requiredgitversion
machinejqversion machinegccversion requiredgccversion machinecplusplusversion
requiredcplusplusversion machineunzipversion requiredunzipversion
machinepythonversion requiredpythonversion machinepipversion requiredpipversion
machinedockerversion dockerstatus user usergroup password ciuserowner ciusergroup
sshpermissioncheck authorizedkeyspermissioncheck sudoers
checkPackages(){ ## Function to check dependencies
machinejavaversion=`java -version 2>&1` ## Get the version of the java installed.
requiredjavaversion="openjdk"
if [[ $machinejavaversion =~ $requiredjavaversion ]] ## If the java version fits to the regex for a valid java version, then
printf "PASS: Java is installed in your machine.\n" ## Then print success message
else
printf "FAILED: Java does not appear to be installed, please install java and try running this again.\n" ## Else print an error
fi
machinegitversion=$(git --version 2>&1) ## Get the version of the git installed.
requiredgitversion="git version"
if [[ $machinegitversion =~ $requiredgitversion ]] ## If the git version fits to
the regex for a valid git version
then
printf "PASS: git is installed in your machine.\n" ## Then print success
message
else
printf "FAILED: git does not appear to be installed, please install git and try
running this again.\n" ## Else print an error
fi
machinejqversion=`jq -r '.test' <<< '{"test": "success"}' 2>> errors.txt` ##
Parse a simple JSON input to output "success", pipe all errors to dev null
if [[ $machinejqversion == "success" ]] ## If the test result in the output of
"success"
then
printf "PASS: JQ is installed in your machine.\n" ## Then print success message
else
printf "FAILED: JQ does not appear to be installed, please install JQ and try
running this again.\n" ## Else print an error
fi
machinegccversion=`gcc --version 2>&1` ## Get the version of the gcc installed.
requiredgccversion="gcc"
if [[ $machinegccversion =~ $requiredgccversion ]] ## If the gcc version fits to
the regex for a valid gcc version
then
printf "PASS: gcc is installed in your machine.\n" ## Then print success message
else
printf "FAILED: gcc does not appear to be installed, please install gcc and try running this again.\n" ## Else print an error
fi
machineunzipversion=`unzip -v 2>&1` ## Get the version of the unzip installed.
requiredunzipversion="UnZip"
if [[ $machineunzipversion =~ $requiredunzipversion ]] ## If the unzip version fits to the regex for a valid unzip version
then
printf "PASS: unzip is installed in your machine.\n" ## Then print success message
else
printf "FAILED: unzip does not appear to be installed, please install unzip and
try running this again.\n" ## Else print an error
fi
machinecplusplusversion=`g++ --version 2>&1` ## Get the version of the c++ installed.
requiredcplusplusversion="g++"
if [[ $machinecplusplusversion =~ $requiredcplusplusversion ]] ## If the c++
version fits to the regex for a valid c++ version
then
printf "PASS: c++ is installed in your machine.\n" ## Then print success message
else
printf "FAILED: c++ does not appear to be installed, please install c++ and try running this again.\n" ## Else print an error
fi
machinepythonversion=`python3 --version 2>&1` ## Get the version of the python installed.
requiredpythonversion="3."
if [[ $machinepythonversion =~ $requiredpythonversion ]] ## If the python version
fits to the regex for a valid python version
then
printf "PASS: Python3 is installed in your machine.\n" ## Then print success message
else
printf "FAILED: Python3 does not appear to be installed, please install python3 and try running this again.\n" ## Else print an error
fi
machinepipversion=`pip3 --version 2>&1` ## Get the version of the pip installed.
requiredpipversion="python3."
if [[ $machinepipversion =~ $requiredpipversion ]] ## If the pip version fits to the regex for a valid pip version
then
printf "PASS: pip3 is installed in your machine.\n" ## Then print success message
else
printf "FAILED: pip3 does not appear to be installed, please install pip3 and try running this again.\n" ## Else print an error
fi
}
checkDocker() { ## Function to check Docker and portainer installation
machinedockerversion=`docker --version` ## Get the version of the docker installed.
requireddockerversion="Docker"
if [[ $machinedockerversion =~ $requireddockerversion ]] ## If the docker version fits to the regex for a valid docker version
then
printf "PASS: docker is installed in your machine.\n" ## Then print success message
else
printf "FAILED: docker does not appear to be installed, please install docker and try running this again.\n" ## Else print an error
fi
dockerstatus=`sudo docker ps` ## Get the list of containers; to execute this command docker should be up
if [[ $dockerstatus =~ "portainer" ]] ## If list of docker containers contains portainer
then
printf "PASS: portainer is installed in your machine.\n" ## Then print success message
else
printf "FAILED: portainer does not appear to be installed, please install portainer and try running this again.\n" ## Else print an error
fi
}
checkUser(){ ## Function to check ciuser creation
user=$(getent passwd ciuser) ## Get ciuser details
if [[ $user =~ "ciuser" ]] ## Check if ciuser is present
then
printf "PASS: ciuser is created.\n" ## Then print success message
else
printf "FAILED: ciuser does not appear to be created, please create ciuser and try running this again.\n" ## Else print an error
fi
if [[ $user =~ "/home/ciuser" ]] ## Check if ciuser's home directory is set
then
printf "PASS: home directory is created for ciuser.\n" ## Then print success message
else
printf "FAILED: It looks like home directory is not set for ciuser, please set ciuser's home directory and try running this again.\n" ## Else print an error
fi
usergroup=$(id ciuser) ## Get the user and group id information
if [[ $usergroup =~ "docker" ]] ## If ciuser is part of docker group
then
printf "PASS: ciuser is present in docker group.\n" ## Then print success message
else
printf "FAILED: ciuser does not appear to be added in docker group, please add ciuser and try running this again.\n" ## Else print an error
fi
password=$(sudo passwd --status ciuser) ## Get ciuser account status information
if [[ $password =~ "Password set" ]] ## Check password status of ciuser
then
printf "PASS: password set successfully for ciuser.\n" ## Then print success message
else
printf "FAILED: ciuser password does not appear to be set, please set ciuser password and try running this again.\n" ## Else print an error
fi
}
checkOS(){ ## Function to check OS
os=$(cat /etc/os-release) ## Get OS information
if [[ $os =~ "Amazon Linux" ]] ## Check for Amazon Linux OS
then
printf "PASS: You have Amazon Linux OS installed.\n" ## Then print success message
else
printf "FAILED: You don't have Amazon Linux installed.\n" ## Else print an error
fi
}
checkPermissions(){ ## Function to check permissions
ciuserowner=$(sudo ls -ld /home/ciuser | awk '{print $3}') ## Get owner information
ciusergroup=$(sudo ls -ld /home/ciuser | awk '{print $4}') ## Get group information
if [[ $ciuserowner == "ciuser" && $ciusergroup == "ciuser" ]] ## Check if ciuser is owner of group ciuser
then
printf "PASS: ciuser is owner.\n" ## Then print success message
else
printf "FAILED: ciuser does not appear to be owner of the ciuser group, please make ciuser owner and try running this again.\n" ## Else print an error
fi
sshpermissioncheck=$(sudo ls -ld /home/ciuser/.ssh | awk '{print $1}') ## Get .ssh file permission
if [[ $sshpermissioncheck == "drwx------" ]] ## Check if permission is 700
then
printf "PASS: .ssh file permission set successfully.\n" ## Then print success message
else
printf "FAILED: It appears that .ssh file permission does not set properly, please set .ssh file permission and try running this again.\n" ## Else print an
error
fi
authorizedkeyspermissioncheck=$(sudo ls -ld /home/ciuser/.ssh/authorized_keys |
awk '{print $1}') ## Get authorized_keys file permission
if [[ $authorizedkeyspermissioncheck == "-rw-------" ]] ## Check if permission is 600
then
printf "PASS: authorized_keys file permission set successfully.\n" ## Then print success message
else
printf "FAILED: It appears that authorized_keys file permission does not set properly, please set authorized_keys file permission and try running this again.\n"
## Else print an error
fi
sudoers=$(sudo cat /etc/sudoers | grep 'ciuser ALL=(ALL) NOPASSWD:ALL') ## Get sudoers file content
if [[ $sudoers == "ciuser ALL=(ALL) NOPASSWD:ALL" ]] ## check for sudo permission
then
printf "PASS: ciuser has sudo permissions.\n" ## Then print success message
else
printf "FAILED: It appears that ciuser doesn't have sudo permissions, please give sudo permission to ciuser and try running this again.\n" ## Else print an
error
fi
}
## Calling functions
checkOS
checkPackages
checkDocker
checkUser
checkPermissions
The successful execution of the script shows that all the dependencies are properly configured. If the script fails and shows that some dependencies are not properly configured, install the missing dependencies, and execute the script again.
Create AMI from EC2 machine
a. Stop the EC2 Instance on which you installed all the packages.
b. Create an AMI of the EC2 Instance.
c. Get the AMI ID of the AMI that you just created.
Copyusermod -aG docker ciuserThis is required so that you don’t need to preface the Docker command with sudo.
-
To automate the continuous building, testing, and deployment of your apps through Jenkins from within the Lazsa Platform, you must provide the connection details of your active Jenkins user account as explained in the following steps:
- Sign in to the Lazsa Platform and click Configuration in the left navigation pane.
- On the Platform Setup screen, on the Cloud Platform, Tools & Technologies tile, click Configure.
- On the Cloud Platform, Tools & Technologies screen, in the DevOps CI/CD Pipeline Configuration section, click Configure.
(After you save connection details for at least one Jenkins user account, you see the Modify button here.)
- On the DevOps CI/CD Pipeline Configuration screen, click the Jenkins tile to configure the connection details of your active Jenkins user account.
- To save the connection properties of your Jenkins account, provide the following details:
Configuration Name: Give a local name to your configuration. Your Jenkins server connection details are saved by this name in the Lazsa Platform.
Jenkins URL: Provide your Jenkins server URL.
Depending on how you want your Jenkins credentials to be managed, do one of the following:
Connect using Lazsa Orchestrator Agent:
Turn on this toggle to use Lazsa Orchestrator Agent to programmatically resolve the Jenkins credentials stored in your secrets management tool within your private network and to establish communication with the Jenkins instance.
In the Lazsa Orchestrator Agent dropdown list, all your configured agents are displayed. Select the one you want to use to connect to your Jenkins instance.
The secrets management tool that the selected Orchestrator Agent is authorized to access for retrieving secrets is auto-selected. Specify the details of Jenkins secrets that the agent should retrieve from the secrets management tool. Currently, we support AWS Secrets Manager and Azure Key Vault.
For AWS Secrets Manager, provide the secret name, username key, and password or token key for your Jenkins account credentials.
For Azure Key Vault, provide the vault name, username secret, and password or token secret for your Jenkins account credentials.
Select Secret Manager
If you don't use the Lazsa Orchestrator Agent, you can directly provide your Jenkins username and password or authentication token in the configuration, or retrieve it programmatically from a secrets management tool of your choice (such as AWS Secrets Manager or Azure Key Vault). Do one of the following:
- Select Lazsa and type your Jenkins account username and password or API token. In this case, your Jenkins user credentials are securely stored in the Lazsa-managed secrets store.
- Select AWS Secrets Manager. In the Secrets Management Tool dropdown list, the AWS Secrets Manager configurations that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen are listed for selection. Select your desired secrets management tool configuration. Provide the secret name, username key, and the password or token key for the Lazsa Platform to retrieve the secrets for your Jenkins account.
- Select Azure Key Vault. In the Secrets Management Tool dropdown list, the Azure Key Vault configurations that you save and activate in the Secret Management section on the Cloud Platform, Tools & Technologies screen are listed for selection. Select your desired secrets management tool configuration. Provide the vault name, username secret, and password or token secret for the Lazsa Platform to retrieve the credential values.
Add Jenkins jobs to run from Lazsa
You can trigger your existing Jenkins job builds, pass parameters, and retrieve job status and results from within the Lazsa Platform. Simply turn on the Add Jenkins jobs to run from Lazsa toggle and add your Jenkins job details and the required parameters in this section. You can then run your configured Jenkins jobs from the Deploy phase of your product development cycle.
To make your Jenkins job management easier and for reusability, you can store the values of your job parameters in the Lazsa Platform. However, if you mark a parameter as sensitive, its value is not stored in the platform. You must enter the values of sensitive parameters when you trigger a job from the Deploy phase.
- Secure configuration details with a password
To password-protect your Jenkins account connection details, enable the Secure configuration details with a password option, enter a password, and then retype it to confirm.This is optional but recommended. When you share the connection details with multiple users, password protection helps you ensure authorized access to the connection details.
Test Connection
Click Test Connection to check if you can connect to the configured Jenkins server successfully.After you save and activate the configured connection details, you can see your saved configuration listed in the DevOps CI/CD Pipeline Configuration section on the Cloud Platform, Tools & Technologies screen.
With this, your Jenkins connection details are successfully configured in the Lazsa Platform. Now, you may want to configure the connection details of other tools involved in your PDLC.
| Recommended Topics
|
What's next? Configure Kubernetes Cluster Connection Details |