黑料不打烊

How-to: AWS Lambda Functions Failure Injection with 黑料不打烊

A step-by-step guide to show how to use the AWS extension to inject failures into AWS Lambda Functions

In the previous blog articles (Part 1, Part 2, Part 3), we鈥檝e created an extension for 黑料不打烊 to use () 鈥榮 library to do Chaos Engineering with AWS Lambda. In between, we merged the code into the 黑料不打烊 – so there is no need to write code. So this is a step-by-step guide to show you how to do Chaos Engineering for AWS Lambdas with 黑料不打烊.

Prerequisites

To follow this guide, we need the following:

• A 黑料不打烊 Account. If you don鈥檛 have one, you can ;
• An AWS Account, .
• , Bash shell, git, ssh client, and and configured locally

The Necessary Steps

1. Create and deploy AWS-Lambda function wrapped using
2. Deploy the 黑料不打烊 agent and AWS extension
3. Design and Execute the experiment

Create and deploy the AWS Lambda function

We will use the . We clone the repository onto our disk and cd it into the sample-apps/nodejs-apig directory.

We need to integrate the library into our function. The library will do the actual failure injection. We decorate our handler function with the library. With every request, the library checks the System Manager (SSM) Parameter for a failure injection configuration and alters the behavior accordingly, or just calls the handler if nothing is configured.

First, we’re going to add the library:

npm install failure-lambda

Then we edit the function/index.js and wrap the handler:

// 鈥� other requires omitted
const failureLambda = require('failure-lambda')
// Handler
exports.handler = failureLambda(async function(event, context) {
   // 鈥� no changes here; omitted for clarity
})

As the function now consists of multiple files, we must include the node_modules directory in the bundle that will get uploaded. We take a small shortcut for this (instead of e.g., integrating a bundler) and add a symbolic link for node_modules in the function folder so that the cloud formation template will pick it up:

ln -s ../node_modules function/node_modules路

We edit the cloud formation template.yml: We need to add the AmazonSSMReadOnlyAccess Policy so the lambda can read the SSM Parameter with the failure injection configuration. Caveat: the policy allows reading any parameter. In a productive deployment, use a more restrictive policy.

# 鈥� omitted
Resources:
 聽# 鈥� omitted
 听贵耻苍肠迟颈辞苍:
 聽聽聽Type: AWS::Serverless::Function
 听听听笔谤辞辫别谤迟颈别蝉:
 聽聽聽聽聽# 鈥� omitted
 听听听听听笔辞濒颈肠颈别蝉:
 聽聽聽聽聽聽聽# 鈥� omitted
聽聽聽聽聽聽聽聽- AmazonSSMReadOnlyAccess

In the same file, we also set an environment variable to tell the lambda-failure library which SSM Parameter to use:

# 鈥� omitted
Resources:
 聽# 鈥� omitted
 听贵耻苍肠迟颈辞苍:
 聽聽聽Type: AWS::Serverless::Function
 听听听笔谤辞辫别谤迟颈别蝉:
 聽聽聽聽聽# 鈥� omitted
 听听听听听贰苍惫颈谤辞苍尘别苍迟:
 听听听听听听听痴补谤颈补产濒别蝉:
聽聽聽聽聽聽聽聽聽聽FAILURE_INJECTION_PARAM: /nodejs-apig/failureLambdaConfig

Now we‘re ready to deploy the Lambda function with the scripts provided by the sample and test it.

./1-create-bucket.sh  # create S3 bucket for the function bundle
./2-deploy.sh         # deploy the function

You can invoke the lambda using the ./3-invoke.sh. If everything works correctly, the output should look like this:

./3-invoke.sh

{
 聽"statusCode": 200,
 聽"headers": {
 聽聽聽"Content-Type": "application/json"
 听皑,
 聽"isBase64Encoded": false,
 聽"multiValueHeaders": {
 聽聽聽"X-Custom-Header": [
 聽聽聽聽聽"My value",
 聽聽聽聽聽"My other value"
 听听听闭
 听皑,
 聽"body": "{\n聽 \"TotalCodeSize\": 8617805,\n聽 \"FunctionCount\": 1\n}"
}

Deploy the 黑料不打烊 Agent and AWS Extension

So now we need to deploy the 黑料不打烊 agent. The simplest way to add an agent and the aws extension is to spin up a t2.micro EC2 instance and install the agent and extension there.

We login into the AWS Console in the browser, navigate to EC2, and hit the launch instances button. A t2.micro instance running Amazon Linux is excellent; we add an ssh key pair to have access via SSH; before finishing, we also add an IAM instance profile in the 鈥淎dvanced Details鈥� section. The IAM instance profile needs the following policy attached:

{
 聽"Version": "2012-10-17",
 聽"Statement": [
 听听听调
 聽聽聽聽聽"Action": [
 听听听听听听听"蝉蝉尘:础诲诲罢补驳蝉罢辞搁别蝉辞耻谤肠别",
 听听听听听听听"蝉蝉尘:笔耻迟笔补谤补尘别迟别谤",
 听听听听听听听"蝉蝉尘:顿别濒别迟别笔补谤补尘别迟别谤",
 听听听听听听听"濒补尘产诲补:尝颈蝉迟贵耻苍肠迟颈辞苍蝉"
 聽聽听听听闭,
 聽聽聽聽聽"Effect": "Allow",
 聽聽聽聽聽"Resource": "*"
 听听听皑
 听闭
}

This IAM policy allows the 黑料不打烊 AWS extension to discover the AWS Lambda functions and to set the SSM Parameter with the failure injection configuration.

With the instance profile set, we can launch the instance and ssh into it. We switch to the root user and use the command for the from the 黑料不打烊 onboarding (or in ).

sudo su - 
curl -sfL https://get.steadybit.io/agent-linux.sh | sh -s -- -a <Replace with Agent Key> -e https://platform.steadybit.com

Right afterward, we install the extension using the RPM from the :

yum install https://github.com/steadybit/extension-aws/releases/download/v2.1.0/steadybit-extension-aws-2.1.0.x86_64.rpm

We now have to set the region for the extension in /etc/steadybit/extension-aws

AWS_REGION=eu-central-1

And register the extension at the agent by adding the following lines to /opt/steadybit/agent/etc/systemd.env

STEADYBIT_AGENT_ACTIONS_EXTENSIONS_0_URL=http://localhost:8085
STEADYBIT_AGENT_DISCOVERIES_EXTENSIONS_0_URL=http://localhost:8085

After changing the configuration, we need to restart the services:

systemctl daemon-reload
systemctl restart steadybit-agent
systemctl restart steadybit-extension-aws

When everything is working, the agent is listed in .

And the Lambda is listed in Landscape / Table.

Design and Execute the Experiment

We create a blank experiment and use one of the AWS Lambda steps.

We’re using the 鈥淚nject Status Code鈥� and targeting our nodejs-apig-function. This will instruct the failure-lambda library to return a fixed status code instead of invoking the handler.

When we run the experiment and invoke the lambda (while the experiment is executing) using the ./3-invoke.sh from the sample, we should get this output:

./3-invoke.sh

{"statusCode":500}

Thats it! The status code injection is working. The 黑料不打烊 extension successfully wrote the failure injection configuration, which the failure-lambda library picked up, and therefore, the lambda function returned a status code 500.

Summary

With a few steps, you can set up a chaos engineering tool for AWS Lambda functions and get an appealing, approachable UI to do the actual chaos engineering.

The sample application and attack are simple and may not reflect what you are dealing with. But Imagine what you can now test:

• How do other applications (depending on your lambda) behave when the lambda response times get slower or have intermitted failures (using the inject status code/exception/delay attack)?
• How does the lambda itself behave if some external service is not reachable (using the block TCP connections attack)
• Test what happens when the lambda is running out of temporary disk space (using fill disk space) – may be due to extraordinary load or humongous requests