Welcome to the CI/CD Boot also known as Vanilla Pipeline

This repository contains the infrastructure as code to bootstrap your next CI/CD project. It is developed with security best practices in mind, provides a robust and automated deployment process that utilizes a combination of CDK Stacks primarily written in TypeScript. It also deploys sample Python Lambda functions. The code is structured in the following way (check the comments, they are important for the future updates):

├── bin
├── config
├── docs
├── lib
│   ├── cdk-pipeline
│   │   ├── app ### here you need to initialize the cdk-stacks you add in the lib/stacks/app
│   │   └── core ### avoid doing changes here in order to keep the update path
│   └── stacks
│       ├── app ### here you define new CDK Stacks which you want to deploy
│       └── core ### avoid doing changes here in order to keep the update path
├── scripts
│   └── tests ### extend and add new
├── src
│   ├── codebuild
│   ├── lambda-functions
│   │   └── test ### extend and add new functions as you see fit
│   └── lambda-layer
│       └── common ### add more packages to requirements.txt
├── test
└── utils

Local Environment Requirements

You need to have the following dependencies in place:

• AWS Account (RES/DEV/INT/PROD)
• Mac OS / Cloud9 with Ubuntu Server 22.04 LTS Platform in RES Account
• Bash/ZSH terminal
• Docker version 24.0.x
• aws-cli v2 here
• AWS credentials and profiles for each environment under ~/.aws/config here
• Node v18.17.* && NPM v10.2.*
• Python >= 3.11
• Pipenv 2023.* here
• jq command line JSON processor jq-1.5

Deployment Architecture

Components

AWS CDK (Cloud Development Kit) scripts to create:

• KMS CMK keys used for the storage encryption
• A CodeBuild and a CodePipeline project to automate the deployment process
• A sample Lambda Stack which includes also a Lambda Layer for manylinux2014_x86_64 platform with some of the most common python libraries, e.g: openssl, pandas and confluent-kafka which shows how to deploy a hello world Lambda function from CDK

Getting started

In order to use CDK to deploy the CICD Boot you need to first install CDK. Instructions how to install can be found here For a more detailed and in-depth explanation on what CDK is and how to use it with TypeScript please refer to this workshop here

Initial Deployment Requirements

Github

To use the CICD Boot for repositories hosted on GitHub you need to establish an AWS CodeStar Connection and use the arn of that connection the next steps.

The existing AWS CodeStar Connections can be checked on the Console or in the CLI with the following command:

aws codestar-connections list-connections --profile ${RES_ACCOUNT_AWS_PROFILE}

If you do not have the AWS CodeStar connection ready please check on how to configure it here.

Optional: Determine VPC and Proxy settings for your pipeline

By default, the Pipeline is configured to run without a VPC. To run inside a VPC, there are two options: VPC and VPC_FROM_LOOK_UP. These options are configured using scripts/configure.sh described in the next section.

Use VPC if you want a single, self-contained pipeline running in a VPC. Not recommended for use with multiple code pipelines in the same account. The VPC is created using defaulted settings.

Use VPC_FROM_LOOK_UP to look up an existing VPC based on its vpc ID. It is recommended to create this VPC prior to deploying the pipeline. Multiple deployments of the pipeline can share the same VPC. If you want to store this vpc ID into SSM then you can do so by prefixing it with resolve:ssm: e.g: resolve:ssm:my-ssm-parameter where the my-ssm-parameter is the string SSM parameter name that contains the vpcId value.

Note: Switching between VPC options may require a complete tear down and redeploy of the pipeline

Proxy Configuration requires proxy information to be stored in Secrets manager. Make note of the secret arn is needed in the next step.

Prepare the local Environment

Prepare local environment with the help of the CICD Boot Setup tool. Start by running the ./scripts/configure.sh command and follow the instructions. Check in the Appendix for detailed instructions.

After you modify the placeholders in the script, make it executable and source those variables:

chmod +x export_vars.sh
source export_vars.sh

Install all the required npm dependencies for CDK:

npm ci ### it installs the frozen dependencies from package-lock.json

Required

Amazon S3 server access logging is a security best practice and should be always enabled. The compliancy logs configuration gets applied to S3 buckets in the bin/aspects.ts for each stage RES/DEV/INT. If you already have existing buckets for compliance logs then set their names in complianceLogBucketName property in the AppConfig.

Make sure that the destination bucket has policy granting s3:PutObject permissions to the logging service principal logging.s3.amazonaws.com (see documentation).

If you do not have compliance log buckets, then the CICD Boot will automatically create them outside of the scope of your current project. The S3 Buckets will be named as configured in the AppConfig.ts

Bootstrap stages

Prepare the RES stage

npm run cdk bootstrap -- --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} aws://${ACCOUNT_RES}/${AWS_REGION}

Prepare the DEV stage

npm run cdk bootstrap -- --profile $DEV_ACCOUNT_AWS_PROFILE  --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust ${ACCOUNT_RES} aws://${ACCOUNT_DEV}/${AWS_REGION}

Prepare the INT stage

npm run cdk bootstrap -- --profile $INT_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust ${ACCOUNT_RES} aws://${ACCOUNT_INT}/${AWS_REGION}

Prepare the PROD stage

*The code below won't run by copy/paste, please update the variables accordingly. *Make sure to also update scripts/warming.sh with the information for the ACCOUNT_PROD same as used for DEV/INT. *Make sure to check also the AppConfig.ts on how to enable PROD stage as there you also need to export more environment variables which are then used to propagate the ACCOUNT_PROD to the CDK Pipeline.

npm run cdk bootstrap -- --profile prod --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust resources_account_id aws://prod_account_id/your_aws_region

Define a new stage

You can create new stages other than the RES, DEV, and INT.

Create environment variables

It is recommended to create environment variables for the new stage in the export_vars.sh file as it is created for the RES, DEV, and INT stages. As an example let us introduce the new PRE_PROD stage that will be present a new pre Production account.

You can edit the export_vars.sh file with your favorite editor or run the following command:

echo "export ACCOUNT_PRE_PROD=<your account id>;" >> export_vars.sh;
echo "export PRE_PROD_ACCOUNT_AWS_PROFILE=<your aws profile>;" >> export_vars.sh;

After you modify the placeholders in the script, source those variables:

source export_vars.sh

Bootstrap the account

The account of the new stage must be bootstrapped as the other accounts and has to trust in the RES account.

npm run cdk bootstrap -- --profile $PRE_PROD_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER} --cloudformation-execution-policies \
arn:aws:iam::aws:policy/AdministratorAccess \
--trust ${ACCOUNT_RES} aws://${ACCOUNT_PRE_PROD}/${AWS_REGION}

Extend the pipeline

In order to the pipeline use the new stage it needs to be added to the deployments definition of the PipelineStack in the bin/app.ts. The order of the stages is controlled by the position in the deployments definition. The account number of the stage can be retrieved from the environment variable, that has been created in the export_vars.sh, and added to the config/AppConfig.ts. The compliance log bucket has to defined as well in the config/AppConfig.ts.

Example config/AppConfig.ts:

const deploymentAccounts = {
  RES: Environment.getEnvVar('ACCOUNT_RES'),
  DEV: Environment.getEnvVar('ACCOUNT_DEV'),
  INT: Environment.getEnvVar('ACCOUNT_INT'),
  PRE_PROD: Environment.getEnvVar('ACCOUNT_PRE_PROD'),
};

export const AppConfig: IAppConfig = {
    ...
    complianceLogBucketName: {
        RES: `compliance-log-${deploymentAccounts.RES}-${region}`,
        DEV: `compliance-log-${deploymentAccounts.DEV}-${region}`,
        INT: `compliance-log-${deploymentAccounts.INT}-${region}`,
        PRE_PROD: `compliance-log-${deploymentAccounts.PRE_PROD}-${region}`,
    },
    ...
}

Then the bin/app.ts can be updated to use the newly created configuration.

new PipelineStack(app, `${AppConfig.applicationName}PipelineStack`, {
  env: { account: AppConfig.deploymentAccounts.RES, region: AppConfig.region },
  applicationName: AppConfig.applicationName,
  applicationQualifier: AppConfig.applicationQualifier,
  logRetentionInDays: AppConfig.logRetentionInDays,
  deployments: {
    RES: { account: AppConfig.deploymentAccounts.RES, region: AppConfig.region },
    DEV: { account: AppConfig.deploymentAccounts.DEV, region: AppConfig.region },
    INT: { account: AppConfig.deploymentAccounts.INT, region: AppConfig.region },
    PRE_PROD: { account: AppConfig.deploymentAccounts.PRE_PROD, region: AppConfig.region },
  },
  ...
}

Deploy all the stacks

npm run cdk deploy -- --all --region ${AWS_REGION} --profile $RES_ACCOUNT_AWS_PROFILE --qualifier ${CDK_QUALIFIER}

Once the command finishes there will be the following CDK Stacks deployed into your RES Account:

• PipelineRepository -> Responsible for either:
  • creating the CodeCommit repository (if you have it selected in config/AppConfig.ts) and the PullRequest automation behind CodeCommit to support scanning with CodeGuru and also run a set of configured commands same as the pipeline does in the Build phase.
  • establishing the CodeStar connection between your AWS RES Account and the configured Github repository.
• SSMParameterStack -> Responsible for creating the parameters in the SSM Parameter Store, e.g: Account IDs.
• VPCStack -> Responsible for enabling the running of the build stages of the pipeline in a VPC, with or without a proxy. By default this stack is not going to be created unless configured via configure.sh. Check here for more information on the possible configurations.
• EncryptionStack -> Responsible for creating the KMS Key used to encrypt all the created CW Log Groups.
• PipelineStack -> Responsible for creating the CodeCommit Repository and the CodePipelineStack with all the CodeBuild Steps

In the other configured stages (DEV/INT/PROD/...) there will be the following example stacks deployed:

• LambdaStack -> Deploys an example Lambda function which comes bundled with an example Lambda Layer containing the specified dependencies in the requirements.txt
• S3BucketStack -> Deploys an example S3 bucket which enforces KMS encryption and Rete