Building CodePipelines with Cloudformation: What's my configuration?

My company started using AWS Codepipeline as a somewhat reluctant PoC. It's not a full featured CICD service, but it is incredibly cost effective and easy to get started with. Amazon's recent release of invoking Lambda functions makes it much more flexible.

We've been using Codepipeline for several months now, and with it starting to look like a longer term solution for us some of the AWS Console limitations are becoming prohibitive. For example you can't move an action around in a stage in the console. Your only option it to delete and recreate the action where you wanted it to be.

Fortunately, most of these struggles are solved by creating your Pipelines in Cloudformation!



The AWS Cloudformation Template reference will get you started with the basics, but it leaves a lot of question marks around the different Action provider types and how to format stages and actions.

For example, we have an Action that deploys a Cloudformation stack. I had the action definition below:

            -
              Name: Deploy-Stack
              InputArtifacts:
                - 
                  Name: our-cft
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: Cloudformation
              Configuration:
                ActionMode: CREATE_UPDATE
                StackName: Our-Staging-Stack
                TemplatePath: our-cft::our-template-name.yml
                TemplateConfiguration: our-cft::stackConfig.json
                Capabilities: CAPABILITY_NAMED_IAM
                RoleArn: arn:aws:iam::/RoleName
                RunOrder: 2

And I was getting a very vague error about the Cloudformation provider not being available. I opened a case with AWS Support only to find out that it was a casing issue, and the Provider needed to be "CloudFormation".

Which was helpful, but still left me with a lot of questions like, "What casing to I need to use for Codebuild? Codedeploy? What are the Configuration items for Invoking Lambda functions?"

After spending some time searching the internet, and finding painfully few examples, I turned to my trusty friend the AWS Powershell Tools. My guess is that we can figure out what values to put into our Cloudformation template by digging out what an existing pipeline has for configuration.

It turns out the Powershell tools have a number of commandlets for interacting with Codepipeline. The one we'll use today is Get-CPPipeline.

To find out more I ran

get-help get-cppipeline

Which shows the information below for the syntax of the command

So let's give it a shot!

I can already guess the stages property is what I'll be interested in next, so let's pull one out to see what it looks like.

I know that this stage has a Cloudformation stack that it deploys, so let's see if I can figure out what the configuration looks like from there. After some digging, I found the ActionTypeId attribute

And there it is! CloudFormation with all the correct capitalization. My guess is confirmed and this seems like a valid plan. I threw together this powershell function to print out information about an existing pipeline. Obviously you'll need to use "Set-AWSCredentials" to pick a profile or IAM account that has access.

function print-pipeline($pipeline) {
    write-host "Pipeline name: $($pipeline.Name)"
    foreach ($stage in $pipeline.stages) {
        write-host "Stage: $($stage.Name)"
        foreach($action in $stage.Actions) {
            Write-host "Action: $($Action.Name)"
            write-host "Action Type ID: $($action.ActionTypeId.Category) - $($action.ActionTypeId.Provider)"
            write-host "Configurations: "
            write-host $action.Configuration
            write-host ""
        }
        Write-host ""
    }

}

set-awscredentials -profilename 
set-defaultawsregion us-east-1

$pipeline = get-cppipeline -name 

print-pipeline $pipeline

A couple notes, the Configuration attribute looks like it's printing out as a list, but I wasn't able to iterate over it. My guess is this is still early enough on in AWS supporting interactions with Codepipeline that some of the functions needed aren't supported.

Lastly, here a some snippets of Action Types we use, pulled out of a couple of our templates.

Using S3 as an artifact source:

        - 
          Name: Source
          Actions:
            - 
              Name: your-action-name
              ActionTypeId: 
                Category: Source
                Owner: AWS
                Version: 1
                Provider: S3
              OutputArtifacts: 
                - 
                  Name: your-output-name
              Configuration: 
                S3Bucket: your-bucket-name
                S3ObjectKey: our/s3/key.zip
              RunOrder: 1

Using AWS Codebuild

          Name: Build
          Actions:
            - 
              Name: YourActionName
              InputArtifacts:
                - 
                  Name: your-source-code
              OutputArtifacts:
                - 
                  Name: your-compiled-code
              ActionTypeId:
                Category: Build
                Owner: AWS
                Version: 1
                Provider: CodeBuild
              Configuration:
                ProjectName: your-code-build-project-name

Invoking a Lambda function

        -
          Name: Test
          Actions:
            -
              Name: your-action-name
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Version: 1
                Provider: Lambda
              Configuration:
                FunctionName: your-function-name
                UserParameters: paraemters-to-pass-in
              RunOrder: 1

Running Opsworks Cookbooks

            -
              Name: RunCookbooks
              InputArtifacts:
                - 
                  Name: your-input-artifact-name
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: OpsWorks
              Configuration:
                DeploymentType: deploy_app
                StackId: your-stack-id
                AppId: your-app-id
              RunOrder: 3

Deploying an app with Codedeploy

            -
              Name: your-action-name
              InputArtifacts:
                - 
                  Name: your-input-artifact
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: CodeDeploy
              Configuration:
                ApplicationName: your-app-name
                DeploymentGroupName: your-group-name
              RunOrder: 4

And lastly, creating a manual approval step

            -
              Name: Approve
              ActionTypeId:
                Category: Approval
                Owner: AWS
                Version: 1
                Provider: Manual
              RunOrder: 1

So far I've been very pleased with the switch. We've been able to re-arrange actions and stages pretty easily, and it made creating additional stages have way fewer clicks.