jenkins>> jenkins_aws_codebuild>> 返回
项目作者： andreistefanciprian

项目描述 ：
trigger aws codebuild projects from Jenkinsfile
高级语言： HCL
项目地址: git://github.com/andreistefanciprian/jenkins_aws_codebuild.git


Description

This is a Jenkins pipeline that builds CodeBuild projects in AWS cloud, then runs these CodeBuild projects.
The CodeBuild projects are building infrastructure within AWS cloud.

The pipeline is doing the following:

  • build AWS resources with terraform (Codebuild projects)
  • parses yaml file with codebuild projects to be executed (python script)
  • starts each of the codebuild projects extracted from yaml file, one by one (python script)

Prerequisites

Have Docker installed. We’ll be running Jenkins on a Docker container.

Spin off a Jenkins docker container with a named volume to preserve jenkins configuration and pipeline for future use:
docker-compose -f jenkins/docker-compose.yaml up --detach

AWS account at https://console.aws.amazon.com/.
AWS access key and secret to be used by Terraform and Jenkins.
AWS Role to be assumed by Terraform and Jenkins.
AWS s3 bucket (terraform backend) and dynamodb table for terraform state lock management.

Create these resources following these steps:

  1. cd prerequisites
  2. terraform init --var-file="../../terraform.tfvars"
  3. terraform plan --var-file="../../terraform.tfvars" -out terraform.tfplan
  4. terraform apply "terraform.tfplan"
  6. # or
  8. cd prerequisites
  9. terraform init --var-file="../../terraform.tfvars"
  10. terraform apply -input=false -auto-approve --var-file="../../terraform.tfvars"

For the steps above, AWS access key and access secret key should be stored in a terraform.tfvars file.
There is a sample with the contents of this file in the main directory of the repository.

Once the prerequsites resources are built, the details of these resources will be shown in the terraform output.
Take these details and populate the related fields in these files:

  • terraform_code/*/main.tf (tfstate s3 bucket and dynamodb table)
  • terraform_code/*/variables.tf (iam role arn)
  • terraform_code/.env (AWS acess key and secret)
  • buildspec.yaml (s3 bucket name)

Configure Jenkins and run pipeline

Go through Jenkins installation steps at: http://localhost:8090.

Define these secrets in Jenkins:

  • aws_access_key: AWS_ACCESS_KEY_ID (secret text); created in prerequsites step
  • aws_secret_key: AWS_SECRET_ACCESS_KEY (secret text); created in prerequsites step
  • aws_region: AWS region (secret text)
  • aws_account: AWS account number (secret text)
  • Git token defined both as secret text and username and password type of secrets (used for git hook and git clone private repo)

AWS credentials inside Codebuild projects:

  • .env file with AWS secrets (AWS_ACCESS_KEY_ID=acces-key and AWS_SECRET_ACCESS_KEY=secret-key generated in prerequsites step) should be made available in s3 bucket (check buildspec.yaml file)
  • terraform used by the CodeBuild projects is running inside a container (check docker-compose.yaml file)
  • the terraform credentials are provided as environment variables via the .env file (check docker-compose.yaml file)

Create Jenkins pipeline job with default settings using Pipeline script from SCM with URL https://github.com/andreistefanciprian/jenkins_aws_codebuild.git.

Run pipeline job!

Destroy resources at the end of this tutorial

  2. # destroy terraform s3 bucket and dynamodb table used for tfstate management
  3. cd prerequisites
  4. terraform destroy --var-file="../../terraform.tfvars"
  6. # destroy AWS resources (AWS creds to be stored in .env file prior to run these commands)
  7. cd terraform_code
  8. make destroy-auto-approve TF_TARGET=infra
  9. make destroy-auto-approve TF_TARGET=static
  10. make destroy-auto-approve TF_TARGET=codebuil
  12. # spin down Jenkins docker container:
  13. docker-compose -f jenkins/docker-compose.yaml down

Other debug commands

Use these AWS CLI commands to manually interact with CodeBuild:

  1. # list CodeBuild projects and builds
  2. aws codebuild list-projects
  3. aws codebuild list-builds
  5. # start CodeBuild project
  6. aws codebuild start-build --project-name codebuildtest-MessageUtil
  7. aws codebuild start-build --project-name newproj-test
  9. # list CodeBuild jobs for specific project
  10. aws codebuild list-builds-for-project --project-name codebuildtest-MessageUtil
  12. # get last build for project
  13. aws codebuild list-builds-for-project --project-name codebuildtest-MessageUtil --query 'ids[0]' --output text
  15. aws codebuild batch-get-builds --ids codebuildtest-MessageUtil:f0682dfe-2d7e-4bec-8061-2008843089e7
  17. # query status of last Codebuild build
  18. build_id=$(aws codebuild list-builds-for-project --project-name codebuildtest-MessageUtil --query 'ids[0]' --output text)
  19. aws codebuild batch-get-builds --ids $build_id --query 'builds[0].buildStatus' --output text

Use these commands to manually check python script:

  1. # create python3 virtual env
  2. python3 -m venv .venv
  4. # activate environment
  5. source .venv/bin/activate
  7. # install requirements
  8. pip install -r requirements.txt
  10. # execute script
  11. aws_account=9238748923565 
  12. python execute_codebuild_from_yaml.py $aws_account

Use these commands to verify you can build resources with terraform from CLI:

  1. TF_VAR_TARGET=static
  2. docker-compose run terraform init $TF_VAR_TARGET
  3. docker-compose run terraform plan -out terraform.tfplan $TF_VAR_TARGET
  4. docker-compose run terraform apply terraform.tfplan
  5. docker-compose run terraform destroy -auto-approve $TF_VAR_TARGET
  7. # using make commands
  8. make deploy-auto-approve TF_TARGET=$TF_VAR_TARGET
  9. make destroy-auto-approve TF_TARGET=$TF_VAR_TARGET