Production Ready Kumologica: AWS Deploy

Deploy Kumologica flow to the AWS account using Kumologica CLI

Photo by Jake Walker on Unsplash

Deployment of Kumologica flow into AWS account from Kumologica Designer is an one click activity. Once logged in to the AWS Account, it is as simple as pressing “Deploy” button. However this is not a solution in CI/CD environments where all activities must be automated. The kumologica sdk release contains cli commands that build binaries and scripts that allows deployment of kumologica flows to cloud platform. In the context of AWS Cloud it is a Cloud Formation script.

The build and deploy process for Kumologica flow is no different than for AWS lambda function, it has 4 steps:

• Build and zip lambda artefacts
• Export Cloud Formation script
• Run AWS Cloud Formation package command
• Run AWS Cloud Formation deploy command

Each of these steps is explained in the following sections.

KL build command

First step is building lambda zip file. To create lambda zip file with kumologica flow run following command:

kl build aws

The build process contains several steps:

Preparation

Build preparation includes creation of ‘build’ subdirectory and copying all project files there.

Flow validation

In this step build command validates flow, for example checks for missing node parameters, presence of event listeners for specific cloud provider ect.

Processing package.json file

Build command will generate aws lambda code and add file information into package.json. It will also analyse dependencies.

Installing npm packages

Then it executes command: npm install -production, this will install all dependencies that are needed.

Creation of zip file

In the last step ‘build’ directory is compressed into zip file.

See kumologica documentation for build command details.

KL export Command

Kumologica cli export command creates complete cloud formation script for Kumologica flow lambda. The created AWS cloud formation script contains not only lambda and iam role artefacts but also all relevant artefacts that lambda depends on, for example api gateway resource, methods and invoke permissions, event bus rules ect. This gives following advantages:

Once kumologica flow is deployed it is fully configured and integrated with aws services, in most cases there is no need to perform manual tasks post deployment to run kumologica flow.

When the cloud formation stack for kumologica flow is deleted, there is no left over artefacts that kumologica flow created, the stack fully cleans up resources for itself.

To export cloud formation script for kumologica flow run command:

kl export cloudformation

This command has superset of aws lambda create-function command parameters, this means it extends the functionality of create lambda function with kumologica specific properties.

One of the most important parameters of kl export command are: --environment and --triggers.

Environment

It is good practice to use environment variables for referencing external resources in lambda function and the same applies to Kumologica flow.

Export Command follows the principle of least privilege. It requires environment variables in order to create iam role with correct resource names/arn used by Kumologica flow.

In cases where resource name is only known during runtime it is necessary to set --strict-mode parameter to false. This will substitute resource name/arn with “*” wildcard.

Triggers

Another important parameter of export command is --triggers. The aws lambda functions are never deployed in isolation, they need to respond to specific event, otherwise they will never be called. Each specific source of event (api gateway, event bus, sqs, …) has its own parameters and other artefacts in order to invoke lambda function.

Below are 2 examples for very common use cases for lambda integrations: api gateway and cron jobs.

Lets assume Kumologica flow has 4 event listeners: 2 for api gateway and 2 AWS CloudWatch events:

Example kumologica flow

API Gateway

Services built in Kumologica and integrated with Amazon Api Gateway require at a minimum attachment to specific api gateway and resource. For example for the Amazon Api Gateway as below:

Example Amazon Api Gateway Before Deployment

Trigger definition looks like:

--triggers: '[{"api": {"apiId": "bpou3xiwza", "parentId": "8egzdb", "resource": "accounts", "stage": "test"}}]'

As a result kumologica cli will create ‘accounts’ resource and attach to the parent resource ‘/customers’ (id: 8egzdb). It will create ANY method and attach Kumologica flow lambda. It will also create {proxy+} resource under ‘/accounts’ resource with ANY method and Kumologica flow lambda attached to it.

The first ANY method defined under /customers/accounts will only handle calls to the /customer/accounts resource. The second ANY method will handle calls to any child resource defined under /customer/accounts, for example GET /customer/accounts/balance.

After deploying Kumologica flow with the example trigger definition, the Amazon API Gateway will look like:

Example Amazon Api Gateway After Deployment

The Amazon Api Gateway trigger properties have more options allowing configuration of authorisers, api key or auth2 scopes.

Cron

Another common use case for Kumologica flows is creation of scheduled invocation of function. This can be achieved using AWS Event bus/AWS Cloud Watch Events and AWS Event rules.

For the example kumologica flow above the triggers will look like:

triggers: '[
{"event": {"expression": "cron(0 1 * * ? *)", "reference": "1am", "name": "CliBuildDemoEvent1am"}}, 
{"event": {"expression": "rate(1 minute)", "reference": "5min", "name": "CliBuildDemoEvent5min"}}
]'

Each Event listener for cron jobs must have reference property set to ‘1am’ and ‘5min’ respectively.

Note that AWS cron expressions are in UTC.

The Event Bus Rules after deploying Kumologica flow:

Below is the version of kl export command including discussed triggers:

kl export cloudformation \
  --zip-file-name build/lambda.zip \
  --bucket-name YOUR_BUCKET_NAME \
  --triggers '[{"api": {"apiId": "bpou3xiwza", "parentId": "8egzdb", "stage": "test", "resource": "accounts"}},  {"event": {"expression": "cron(0 1 * * ? *)", "reference": "1am", "name": "CliBuildDemoEvent1am"}}, {"event": {"expression": "rate(1 minute)", "reference": "5min", "name": "CliBuildDemoEvent5min"}}]'

The remaing 2 commands are aws cli commands: package and deploy.

Package command

The aws cli command to prepare lambda for deployment to aws account. Use the script file created by kl export command as value of --template-file parameter.

aws cloudformation package \
  --template-file template.json \
  --s3-bucket YOUR_BUCKET_NAME \
  --output-template-file template-lambda.yaml 

Deploy command

This is the final aws cli command to deploy lambda into aws account:

aws cloudformation deploy \
--template-file template-lambda.yaml \
--stack-name cli-aws-build-demo \ 
--capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND

Conclusion

Kumologica SDK contains commands that allows deployment of Kumologica flows into cloud provider from ci/cd environments.

It compliments cloud providers tooling and does not force use of any specific frameworks. This article presented use of Kumologica cli with AWS cli and cloud formation scripts and provided examples of creating triggers for AWS Api Gateway and AWS Event Bus/AWS Cloud Watch Events.

Remember Kumologica is free to download and use. Go ahead and give it a try, we would love to hear your feedback.

More information

• The source code to this article is in our kumologica-demos github repository.
• For Kumologica CLI Reference Guide see: Kumologica CLI Reference
• For more information about Kumologica SDK Installation see: Kumologica SDK Installation
• Join our community group on discord
• Visit kumologica.com for information about sdk, designer, documentation, tutorials, support and professional services.