<p>This post is a follow-up to my <a href="/aws-api-gateway-iam.html">authenticated API Gateway post</a>.</p>
<p>I've written a Lambda function and presented it using API Gateway. The API
uses Amazon's AIM keys for authentication. The next step is to write the code
to source control (AWS CodeBuild Git) and have it automatically deploy.</p>
<p><strong>Note</strong>: <a href="https://docs.aws.amazon.com/lambda/latest/dg/build-pipeline.html">This official guide on AWS</a>
is really good, and I copied big parts it. I use Python instead of
JavaScript in this post, and include my own experience while following the
guide.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-an-s3-bucket">Create an S3 Bucket</a></li>
<li><a href="#commit-your-function-to-source-control">Commit your function to source control</a></li>
<li><a href="#define-an-aws-sam-template">Define an AWS SAM template</a>
<ul>
<li><a href="#write-the-template-file">Write the template file</a></li>
<li><a href="#test-the-template">Test the template</a></li>
</ul>
</li>
<li><a href="#create-a-codebuild-project">Create a CodeBuild Project</a>
<ul>
<li><a href="#write-the-buildspecyml">Write the buildspec.yml</a></li>
<li><a href="#create-a-role-for-codebuild">Create a role for CodeBuild</a></li>
<li><a href="#define-the-build-project">Define the build project</a></li>
</ul>
</li>
<li><a href="#create-a-pipeline">Create a Pipeline</a>
<ul>
<li><a href="#create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</a></li>
<li><a href="#create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</a></li>
<li><a href="#create-the-pipeline">Create the Pipeline</a></li>
</ul>
</li>
</ul>
<h1 id="create-an-s3-bucket">Create an S3 Bucket</h1>
<p>Open the <a href="https://s3.console.aws.amazon.com/s3/">S3 Console</a> and create a
bucket to store your build artifacts.</p>
<hr>
<hr>
<h1 id="commit-your-function-to-source-control">Commit your function to source control</h1>
<p>I'm using a dedicated repository on AWS CodeCommit to store the code, and
Python3 for the language. Here's a very simple Lambda function I use for
testing:</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<hr>
<h1 id="define-an-aws-sam-template">Define an AWS SAM template</h1>
<h2 id="write-the-template-file">Write the template file</h2>
<p>This was my first time using AWS SAM, so consider these my observations and not
necessarily fact.</p>
<p>SAM stands for Serverless Application Model.</p>
<ul>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html">SAM Reference Page</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-api.html">API Event type</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">What is SAM</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html">SAM Specification</a></li>
</ul>
<p>The fields I've changed from Amazon's example for this post:</p>
<ul>
<li><strong>Description</strong> - string describing the serverless application. This
description will be attached to the CloudFormation stack.</li>
<li><strong>Resources.[resourceName]</strong> - self-defined resource, my function.
The resource is a Lambda function because it's type is
<code>AWS::Serverless::Function</code></li>
<li><strong>Handler</strong> - Which method will run in the function</li>
<li><strong>Runtime</strong> - The language to be used</li>
<li><strong>CodeUri</strong> - Path to my file defining the function</li>
<li><strong>Resource.[resourceName].Events[eventName]</strong> - Name of the event which
triggers the function.</li>
<li><strong>Resource.[resourceName].Events[eventName].path</strong> - path for the URL</li>
</ul>
<pre><code class="language-yaml"># template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample Continuously deploy function
Resources:
  TestCicdFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: cicd-test.lambda_handler
      Runtime: python3.8
      CodeUri: ./cicd-test.py
      Events:
        TestCicdAPI:
          Type: Api
          Properties:
            Path: /Test
            Method: GET
</code></pre>
<h2 id="test-the-template">Test the template</h2>
<p>Install the AWS SAM CLI application.</p>
<pre><code class="language-bash">pip install --user aws-sam-cli
</code></pre>
<p>Build the package and upload it to your S3 bucket. From your project
root, set your bucket name then run the following.</p>
<pre><code class="language-bash"># Set the bucket name first
BUCKET=''

aws cloudformation package \
  --template-file template.yml \
  --s3-bucket $BUCKET \
  --output-template-file outputtemplate.yml
</code></pre>
<p>The package command will upload your python file to S3 and output a new
template file with the S3 path replacing the <code>CodeUri</code>.</p>
<pre><code class="language-text">Uploading to &#x3C;ID>  381 / 381.0  (100.00%)
Successfully packaged artifacts and wrote output template to file outputtemplate.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /home/vagrant/arcus-lambda/outputtemplate.yml --stack-name &#x3C;YOUR STACK NAME>
</code></pre>
<p>Now deploy it. Since there's no CloudFormation stack created yet, I make a new
one named <code>test-cicd-lambda</code>. Note that in the above command output, the
example they give for deploy is wrong. You need to include the
<code>--capabilities CAPABILITY_IAM</code> argument for this template too.</p>
<pre><code class="language-bash">aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --template-file /home/vagrant/arcus-lambda/outputtemplate.yml \
  --stack-name test-cicd-lambda
</code></pre>
<p>The application is now deployed.</p>
<ul>
<li>Stack is visible in the  <a href="https://ca-central-1.console.aws.amazon.com/cloudformation">CloudFormation console</a></li>
<li>API is created in the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway console</a></li>
<li>Function is created in the <a href="https://ca-central-1.console.aws.amazon.com/lambda/">Lambda console</a>.
The function also has an Application page in the web console now.</li>
</ul>
<hr>
<h1 id="create-a-codebuild-project">Create a CodeBuild Project</h1>
<p>I recently did a whole post on this <a href="/aws-codepipeline-ecr.html">here</a>, so I'm
going to be light on the details. Head over to that post to fill in any blanks.
In that post I was building Docker images and pushing them to ECR. This is
similar, except we push python files to Lambda instead.</p>
<h2 id="write-the-buildspecyml">Write the buildspec.yml</h2>
<p>Put this in the project root, it defines what CodeBuild will do. We'll use an
environment variable for the <code>BUCKET</code> definition.</p>
<p>If you're not sure what to pick for a runtime, check <a href="https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html">this list</a>.</p>
<pre><code class="language-yaml"># buildspec.yml
version: 0.2
phases:
  install:
    runtime-versions:
        python: 3.8
  build:
    commands:
      - aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
artifacts:
  type: zip
  files:
    - template.yml
    - outputtemplate.yml
</code></pre>
<p>Commit the file to source control.</p>
<h2 id="create-a-role-for-codebuild">Create a role for CodeBuild</h2>
<p>I use the role I defined for CodeBuild in my previous post. It's a bit overly
permissive but it works. Other than the policies which get created
automatically, mine has these:</p>
<ul>
<li><code>AWSCodeCommitFullAccess</code></li>
<li><code>AmazonEC2ContainerRegistryFullAccess</code></li>
<li><code>AmazonS3FullAccess</code></li>
<li><code>CloudWatchLogsFullAccess</code></li>
<li><code>AWSCodeBuildAdminAccess</code></li>
</ul>
<h2 id="define-the-build-project">Define the build project</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/">CodeBuild console</a></li>
<li>Create build project</li>
<li>Name the build and give it a description</li>
<li>Source: Add your repository</li>
<li>Environment
<ol>
<li>Managed Image</li>
<li>Operating system: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: the newest one</li>
<li>Environment type: Linux</li>
<li>Privileged: False
<ol>
<li>Service Role: Existing role - choose the one you made above</li>
</ol>
</li>
<li>Additional Configuration - environment variables
<ul>
<li>name: BUCKET</li>
<li>value: Your bucket name</li>
</ul>
</li>
</ol>
</li>
<li>Buildspec: use buildspec file</li>
<li>Artifacts: no artifacts</li>
<li>Logs: set a group name for CloudWatch</li>
<li>Create build project</li>
</ol>
<p>Test the build before moving on and make sure it works.</p>
<hr>
<h1 id="create-a-pipeline">Create a Pipeline</h1>
<h2 id="create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</h2>
<p>Open the IAM console and create a new policy.
Define the policy using the following JSON.
These wildcards will cover everything you need, though it might be a bit
permissive.</p>
<pre><code class="language-json">{
    "Statement": [
        {
            "Action": [
                "apigateway:*",
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}
</code></pre>
<h2 id="create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</h2>
<p>In the role wizard of the IAM page:</p>
<ol>
<li>Create role</li>
<li>Trusted Entity: <code>CloudFormation</code></li>
<li>Permissions:
<ol>
<li><code>AWSLambdaExecute</code></li>
<li>Your new policy created above</li>
</ol>
</li>
<li>Role Name: assign a name</li>
</ol>
<h2 id="create-the-pipeline">Create the Pipeline</h2>
<p>When code is committed, you want CodeBuild to run and re-deploy your serverless
application.</p>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: Create a new service role for the pipeline. AWS will set sane
permissions for it, no special role is needed. I like to have a consistent
naming convention as you end up with a ton of IAM entities after a while.</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Project Name: Choose the project you just made</li>
<li>Next</li>
<li>Deploy Provider: AWS CloudFormation</li>
<li>Action Mode: Create or replace a change set</li>
<li>Stack Name: choose a name. I used the one from the test above.</li>
<li>Change Set Name: choose a name</li>
<li>Template:
<ul>
<li>Artifact name: <code>BuildArtifact</code></li>
<li>File name: <code>outputtemplate.yml</code></li>
</ul>
</li>
<li>Capabilities: <code>CAPABILITY_IAM</code></li>
<li>Role Name: The role you defined for CodeDeploy above</li>
<li>Next</li>
<li>Create Pipeline</li>
</ol>
<p>That should do it. Now whenever you push code to your master branch, the
serverless application will re-deploy.</p>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<h1 id="workflow">Workflow</h1>
<ol>
<li>Commit code to AWS CodeCommit</li>
<li>CloudWatch rule triggers CodePipeline</li>
<li>CodePipeline uses CodeBuild to build the Docker image</li>
<li>CodePipeline pushes the image to ECR</li>
</ol>
<hr>
<h1 id="deploy-code-to-codecommit">Deploy Code to CodeCommit</h1>
<ol>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories">CodeCommit</a>.</li>
<li>Create a repository</li>
<li>Go to your IAM page and edit your user.
<ol>
<li>Look for "HTTPS Git credentials for AWS CodeCommit"</li>
<li>click Generate Credentials</li>
<li>Save your credentials.</li>
</ol>
</li>
<li>Grab the HTTPS URL for your repository and <code>git clone</code> it. Use your new
credentials.</li>
<li>Commit your code (including a Dockerfile) to the CodeCommit repository.</li>
</ol>
<hr>
<h1 id="write-buildspecyml">Write buildspec.yml</h1>
<p>AWS CodeBuild will look for a <code>buildspec.yml</code> file to dictate its actions.
This is where we define how the image will be built. Put it in your CodeCommit
project's root directory.</p>
<p>Here's mine. This will build a docker image and tag it with a timestamp, then
push it to ECR as the timestamp and latest. The variables are defined inside
the build as Environment Variables.</p>
<pre><code class="language-yml"># buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region ca-central-1)
      - IMAGE_TAG=$(date +%Y.%m.%d.%I.%M)
  build:
    commands:
      - echo Build started on `date`
      - echo "Building the Docker image."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
</code></pre>
<hr>
<h1 id="define-the-pipeline">Define the Pipeline</h1>
<h2 id="cloud-build-role">Cloud Build Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-cloudbuild-project">Define CloudBuild project</h2>
<ol>
<li>Navigate to the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/projects">CloudBuild Console</a></li>
<li>Create build project</li>
<li>No build badge</li>
<li>Source provider: CodeCommit</li>
<li>Choose your repository</li>
<li>Choose your branch</li>
<li>Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtime: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: latest</li>
<li>Environment type: Linux</li>
<li>Privileged: true</li>
<li>Service Role: Existing service role. Use the one you just made.</li>
<li>Allow CodeBuild to modify this role: False (we gave it admin)</li>
<li>Additional Configuration - Environment variables (click Add to get more)
- IMAGE_REPO_NAME: (your image name)
- AWS_ACCOUNT_ID: (your numeric AWS account iD)
- AWS_DEFAULT_REGION: ca-central-1</li>
<li>Buildspec: Use a buildspec file</li>
<li>Artifacts: No artifacts</li>
<li>Logs: Give a group and stream name for CloudWatch</li>
<li>Create Build Project</li>
</ol>
<p>Now test the build by clicking "Start build". It should create your image.
You'll need to disable artifacts.</p>
<p>Once the build works and your image is pushed successfully to ECR, you can move
on to setting up a pipeline so this happens automatically.</p>
<h2 id="create-pipeline-role">Create Pipeline Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-the-pipeline-1">Define the pipeline</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: New service role</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Region: Canada (Central)</li>
<li>Project Name: Create Project
<ol>
<li>Project Name: Choose a name</li>
<li>Description: Add a description</li>
<li>Environment image: Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: Always use the latest</li>
<li>Environment type: Linux</li>
<li>Privileged: True</li>
<li>Service Role: New service role</li>
<li>Role Name: default name is fine</li>
<li>Advanced Configuration - Environment variables (click Add to get more)
<ul>
<li>IMAGE_REPO_NAME: (your image name)</li>
<li>AWS_ACCOUNT_ID: (your numeric AWS account iD)</li>
<li>AWS_DEFAULT_REGION: ca-central-1</li>
</ul>
</li>
<li>Build specifications: Use a buildspec file</li>
<li>Buildspec name: Leave it blank so it uses buildspec.yml</li>
<li>CloudWatch logs: True</li>
<li>Group name: pick a name</li>
<li>Stream name: pick a name</li>
<li>S3 logs: false</li>
<li>Continue to CodePipeline</li>
</ol>
</li>
<li>Next</li>
<li>Skip deploy stage, pushing to ECR is enough</li>
<li>Create pipeline</li>
</ol>
<p>This will re-run the build that you tested before. It should pass again.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags