--- title: "Chaos Engineering: Testing Resiliency with Chaos Monkey and Gremlin" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/chaos-engineering-resiliency-testing-monkey-gremlin --- ![Blog post image for Chaos Engineering: Testing Resiliency with Chaos Monkey and Gremlin - How Chaos Engineering builds resilient systems. Chaos Monkey and Gremlin inject faults in AWS and Kubernetes so you find the weak spots before an outage does. Best practices, experiment types, and frequently asked questions for resilient software testing.](/_astro/hero.D_hw4oVT_4MLgR.webp) [Home](/)›[Blog](/blog)›[All Categories](/blog/categories)›[Chaos Engineering](/blog/categories/chaos-engineering) Blog [Chaos Engineering](/blog/categories/chaos-engineering)[System Reliability](/blog/categories/system-reliability)[DevOps](/blog/categories/devops)[Site Reliability Engineering](/blog/categories/site-reliability-engineering)[Testing](/blog/categories/testing) # Chaos Engineering: Testing Resiliency with Chaos Monkey and Gremlin [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 31 Jan 202615 Mins read25 Mins listen [Markdown for AI(opens in a new tab)](/post/chaos-engineering-resiliency-testing-monkey-gremlin/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How Chaos Engineering builds resilient systems. Chaos Monkey and Gremlin inject faults in AWS and Kubernetes so you find the weak spots before an outage does. Best practices, experiment types, and frequently asked questions for resilient software testing. Series [Reliability & Resilience Engineering](/series/reliability--resilience-engineering)1/2 [NextDesigning SLOs and Error Budgets: Your Blueprint for Sustainable Reliability](/blog/post/designing-slos-error-budgets-reliability-blueprint) All posts in this series (2) Blog2 1. [Chaos Engineering: Testing Resiliency with Chaos Monkey and GremlinYou are here](/blog/post/chaos-engineering-resiliency-testing-monkey-gremlin) 2. [Designing SLOs and Error Budgets: Your Blueprint for Sustainable Reliability](/blog/post/designing-slos-error-budgets-reliability-blueprint) ### Chaos Engineering: Testing Resiliency with Chaos Monkey and Gremlin Contents [What exactly is Chaos Engineering?](#what-exactly-is-chaos-engineering)[Why does system resiliency matter today?](#why-does-system-resiliency-matter-today)[What are the guiding principles of Chaos Engineering?](#what-are-the-guiding-principles-of-chaos-engineering)[Chaos Monkey: the original primate of production chaos](#chaos-monkey-the-original-primate-of-production-chaos)[Gremlin: the modern platform for controlled chaos](#gremlin-the-modern-platform-for-controlled-chaos)[Chaos Monkey vs. Gremlin: which tool is right for you?](#chaos-monkey-vs-gremlin-which-tool-is-right-for-you)[How do we inject chaos in AWS environments?](#how-do-we-inject-chaos-in-aws-environments)[References](#references) Modern software systems are incredibly complex. They’re spread across massive networks with countless moving parts. Because of this complexity, unexpected failures are inevitable. Servers crash. Networks slow down. Dependencies fail. Instead of waiting for something to break at 3 AM and scrambling to fix it, there’s a smarter approach: find and fix weaknesses before they cause real problems. That’s what Chaos Engineering is all about. As Dr. Werner Vogels, Amazon’s CTO, famously said, “Everything fails, all the time.” This simple truth is the foundation of an entire field dedicated to preparing systems for inevitable failures. Chaos Engineering is essentially a way to test how strong your system is by intentionally introducing faults. The goal? Find weak spots and fix them before they lead to real outages. It’s not about creating chaos for chaos’s sake. It’s a careful, scientific method that uses controlled experiments to identify and prevent problems before they happen. This approach changes how companies think about system reliability. The old way was reactive: wait for something to fail, then fix it. That’s expensive in terms of downtime, reputation damage, and wasted resources. Chaos Engineering flips this around. It’s proactive. You’re preventing failures instead of just recovering from them. That shift comes out of a more mature engineering culture, one that values prevention over firefighting. ## [What exactly is Chaos Engineering?](#what-exactly-is-chaos-engineering) Chaos Engineering helps you understand how failures happen in complex systems and gives you practical ways to prevent or reduce them. It’s about running controlled experiments to uncover hidden weaknesses. Think of it like a vaccination for your software: you’re introducing a small, controlled problem to build immunity against bigger, real-world disasters. Here’s the key difference between Chaos Engineering and traditional testing. Traditional testing verifies that your system works as expected. It’s checking known features and confirming requirements. You’re testing what you know. Chaos Engineering, on the other hand, looks for unknown weaknesses before they become problems. It’s proactive, not reactive. This shift matters more than you might think. Modern systems run in unpredictable environments with countless variables. Relying only on traditional testing leaves blind spots. You won’t know how solid your system really is until it’s under real pressure. By intentionally introducing controlled failures, you’ll discover which parts of your system are solid and which need work. This organized approach helps companies build stronger products, which directly impacts their bottom line and customer satisfaction. But there’s more to it than just technical benefits. When systems aren’t resilient enough, the hidden costs add up fast. You’re not just losing money during outages. You’re also slowly losing customer trust, burning out your operations team with constant incident response, exhausting developers who are always firefighting, and missing opportunities for innovation because your resources are tied up maintaining basic stability. Chaos Engineering tackles these problems early, creating a more sustainable and productive engineering environment. ## [Why does system resiliency matter today?](#why-does-system-resiliency-matter-today) When we talk about resiliency in software, we’re talking about your system’s ability to keep working even when things go wrong. It might be running in a degraded state, but it’s still keeping core functions alive. More importantly, it’s about how quickly you can get back to normal operations. Real resilience means you can anticipate problems, absorb the impact, adapt to changes, and recover quickly from whatever the environment throws at you. In a hyper-connected world, even brief outages have massive consequences. Think about e-commerce during Black Friday, banking systems processing millions of transactions, or healthcare systems managing patient data. When these systems fail, the damage is immediate and severe. You’re looking at revenue losses, brand damage, frustrated customers, and potential regulatory penalties. The Digital Operational Resilience Act (DORA), for example, requires regular resiliency testing to identify weaknesses. This regulatory pressure isn’t going away. Building resilient systems isn’t optional anymore. It’s essential for protecting your business, your customers, and your reputation. The strategic importance of resilience goes way beyond technical features. It’s a competitive differentiator and, increasingly, a regulatory requirement. System uptime and reliability drive customer satisfaction, revenue, and compliance directly. Companies that invest in resilience with practices like Chaos Engineering are in a better position to meet strict regulations and to outpace competitors with more stable, trustworthy services. System fragility that nobody addresses also carries costs that never appear on an outage invoice. They go beyond the immediate financial hit to include the gradual erosion of customer trust, the operational overhead of constant incident response, developer burnout from continuous firefighting, and the innovation you never get to because resources are tied up maintaining basic stability. Finding and fixing those weaknesses early cuts those long-term, mostly invisible expenses, and it makes for a more sustainable and productive engineering organization. ## [What are the guiding principles of Chaos Engineering?](#what-are-the-guiding-principles-of-chaos-engineering) Chaos Engineering isn’t about randomly breaking things and seeing what happens. It’s a disciplined, scientific approach to understanding how systems behave under stress. These core principles guide how you design and run experiments, so you come away with useful insight instead of accidental damage. 1. **Build a hypothesis around steady state behavior:** Before introducing any disruption, you need to understand what normal looks like. Your steady state is the system’s baseline behavior, measured by key metrics like throughput, error rates, and response times. Once you’ve established this baseline, you form a hypothesis about how the system should behave when you introduce a specific fault. For example: “Even if our payment microservice fails, users can still browse products and add items to their cart.” This baseline matters for measuring the actual impact of your chaos experiments. 2. **Mimic real-world problems:** Your experiments should simulate actual failures that happen in production environments. Then what you learn applies to real incidents. Real-world problems include server crashes, network latency, database slowdowns, sudden traffic spikes, or third-party API failures. The more realistic your simulations, the more valuable your learnings. 3. **Test in production (or production-like environments):** Systems behave differently under real load with real traffic patterns. For the most accurate results, you should run experiments in production or in environments that closely mirror it. Yes, this sounds risky, but you’ll do it with strict safety controls to limit the blast radius. Start small in staging environments to build confidence, then gradually move to production with tight monitoring. The blast radius concept is critical here. By carefully controlling the scope of your experiments, you can isolate variables, observe specific impacts, and learn how your system reacts to particular faults without causing widespread damage. 4. **Automate your chaos tests:** Running experiments manually is time-consuming and doesn’t scale. Automation makes tests run consistently and reliably. The best approach? Integrate chaos testing directly into your CI/CD pipeline. This way, you’re catching problems early during development and deployment, not after they reach production. One thing you absolutely can’t skip: strong monitoring and observability. Without good monitoring, you can’t define your steady state, measure the impact of experiments, or detect when something goes wrong. Investing in thorough monitoring and logging is both a prerequisite and an ongoing requirement for successful Chaos Engineering. There’s another benefit that’s often overlooked. Chaos Engineering makes your team operationally stronger. By regularly simulating failures, teams practice incident response, test their alerting systems, and sharpen their debugging skills. This regular exposure to stress builds confidence and creates a more mature, resilient engineering team. You’re not just finding system bugs. You’re also uncovering and fixing gaps in your operational processes and team readiness. ## [Chaos Monkey: the original primate of production chaos](#chaos-monkey-the-original-primate-of-production-chaos) Chaos Monkey was born out of necessity at Netflix. As they pioneered cloud-native architecture, they faced a critical challenge: keeping their streaming service available while running on thousands of cloud servers. Their solution? Build a tool that randomly shuts down instances in production. Sounds crazy, right? But this seemingly destructive action had a real purpose. It forced engineers to design services that could handle instance failures from day one. The tool exposed engineers to failures frequently, encouraging them to build naturally resilient services. Chaos Monkey’s job is simple: randomly terminate virtual machine instances and containers during specific time windows. While its main action is random termination, you can customize its behavior through configuration files and integration with Spinnaker (Netflix’s continuous delivery platform). It includes an outage checker that prevents it from running during existing incidents, so it won’t make ongoing problems worse. Here’s a basic example of a Chaos Monkey configuration: chaosmonkey.yml ``` 1# Chaos Monkey configuration for AWS2accounts:3 - name: production4 enabled: true5 # Only run during business hours (PST)6 schedule:7 enabled: true8 startHour: 99 endHour: 1710 timezone: America/Los_Angeles11 12terminationStrategy:13 # Randomly terminate instances14 randomSelection: true15 16 # Probability of termination (10% chance)17 probability: 0.118 19 # Maximum number of instances to terminate per run20 maxTerminationsPerDay: 521 22# Exclude critical services23exceptions:24 - serviceName: auth-service25 - serviceName: payment-processor26 27# Notification settings28notifications:29 email:30 - devops@example.com31 slack:32 channel: '#chaos-engineering' ``` Despite its historical significance, Chaos Monkey has some significant limitations. Its biggest constraint is limited attack types. It only does one thing: random instance termination. This seriously limits the kinds of failure scenarios you can simulate. The unpredictable, completely random nature means you have limited control over the blast radius. This lack of precision can cause more harm than good if your system isn’t ready. Chaos Monkey also has major dependencies. It needs Spinnaker and MySQL for full integration. A big downside? Netflix no longer actively develops or maintains it, making it less practical for teams looking for ongoing support and new features. It also lacks built-in recovery or rollback mechanisms. Any fault tolerance or outage detection requires custom code. Chaos Monkey works with environments that Spinnaker supports: AWS, Google Compute Engine (GCE), Azure, and Kubernetes. It’s been specifically tested with AWS, GCE, and Kubernetes. If your applications are managed through Spinnaker, you can set up Chaos Monkey to terminate instances within these cloud and container platforms. The evolution from Chaos Monkey to modern tools shows a shift from forced resilience to controlled learning. Chaos Monkey’s original idea worked because it was simple. By randomly terminating instances, it forced engineers to build resilience into their services. But its limitations (just one random fault type and no ongoing maintenance) show that while random disruption can uncover weaknesses, lasting resilience needs a more controlled, varied, and analytical approach. The industry has moved from a raw “break it and see what happens” mindset to a more mature, controlled, and data-driven experimental science. ## [Gremlin: the modern platform for controlled chaos](#gremlin-the-modern-platform-for-controlled-chaos) Gremlin is a leading cloud-native platform built specifically to make Chaos Engineering safe, easy, and secure. Its main goal? Improve system uptime, validate reliability, and help companies build a strong reliability culture. Unlike Chaos Monkey’s random approach, Gremlin gives you precise control over fault injection. Gremlin provides an extensive fault injection library that lets you simulate real-world failures across different system layers: **Resource attacks:** These test how your system handles resource constraints. - CPU attacks stress test high-demand scenarios - Memory attacks check for leaks or resource-heavy applications - I/O attacks create read/write pressure to test storage performance - GPU attacks stress AI, LLM, and video encoding workloads Here’s a simple example of running a CPU attack with Gremlin: gremlin-cpu-attack.sh ``` #!/bin/bash # Attack all cores on a specific container for 60 secondsgremlin attack-container \ --container-id abc123 \ --type cpu \ --cores 0 \ --length 60 ``` **Network attacks:** These simulate network problems. - Blackhole attacks drop all network traffic to simulate complete outages - Latency attacks inject delays to test responsiveness under slow networks - Packet Loss attacks drop or corrupt traffic to mimic poor network conditions - DNS attacks block DNS access to test fallback mechanisms Here’s how to inject network latency: gremlin-latency-attack.sh ``` #!/bin/bash # Add 100ms latency to all egress traffic for 2 minutesgremlin attack-container \ --container-id abc123 \ --type latency \ --delay 100 \ --length 120 ``` **State attacks:** These test application and system state changes. - Process Killer stops specific processes to simulate application crashes - Shutdown attacks restart the host OS to test host failure recovery - Time Travel attacks change system time to test for clock drift or certificate expiry - Certificate checks verify certificate chains for expiration Here’s a process killer example: gremlin-process-kill.sh ``` #!/bin/bash # Kill all nginx processes and repeat every 5 seconds for 2 minutesgremlin attack-container \ --container-id abc123 \ --type process_killer \ --process nginx \ --interval 5 \ --length 120 ``` You can combine these attack types to create hundreds of pre-built and custom scenarios for very targeted, complex simulations. Gremlin’s platform supports multi-environment deployments. It’s truly cloud-native and runs almost anywhere: all major public clouds (AWS, Azure, GCP), Linux, Windows, containerized environments like Kubernetes, and even on-premise with Gremlin Private Edition. This wide compatibility makes it versatile for companies with diverse infrastructure. Beyond fault injection, Gremlin offers features that improve your Chaos Engineering practice. Its GameDay Manager helps organize reliability events, cutting down prep and execution time. The platform automatically analyzes and stores experiment results, so teams can review outcomes and turn data into real improvements. It integrates with Jira for efficient action item tracking. Gremlin also provides reliability scoring and continuous risk monitoring, so you can define, measure, and track service reliability across your organization. It can automatically discover and test system dependencies, which gives you a deeper view of where the system is weak. Gremlin’s feature set (diverse fault types, multi-cloud support, GameDay management, Jira integration) shows that Chaos Engineering has matured into a complete managed service. That evolution follows the growing complexity of modern systems and the growing need for advanced reliability management. What makes Gremlin particularly valuable is its focus on safety and control. By positioning itself as making Chaos Engineering safe, easy, and secure, Gremlin directly addresses common concerns about the practice. Many teams worry about causing more harm than good. By offering precise control over fault injection, automatic halt and rollback features, and a user-friendly interface, Gremlin turns a potentially risky practice into a controlled and useful one. That changes how willing businesses are to adopt these methods, especially in sensitive production environments. ## [Chaos Monkey vs. Gremlin: which tool is right for you?](#chaos-monkey-vs-gremlin-which-tool-is-right-for-you) Choosing between Chaos Monkey and Gremlin depends on your company’s specific needs, budget, and chaos engineering maturity. While both tools aim to make systems more resilient, their approaches and capabilities are quite different. Here’s a side-by-side comparison: **Feature/Aspect** **Chaos Monkey** **Gremlin** **Origin & Maintenance** Developed by Netflix, historically significant. No longer actively developed or maintained. Commercial product, actively developed and maintained. **Control Over Faults** Injects faults randomly, giving a more realistic test environment for broad resilience. Limited control over blast radius and execution. Offers precise control over fault injection, allowing targeted experiments. Provides automatic halt and rollback mechanisms. **Types of Faults** Primarily one attack type: random instance termination (shutdown). Wide range of fault types: CPU, Memory, Disk, I/O, Blackhole, Latency, Packet Loss, Process Killer, Shutdown, DNS, Time Travel, Certificate Expiry, GPU. **Cloud/Environment Support** Tightly integrated with Netflix OSS and AWS. Works with AWS, GCE, and Kubernetes via Spinnaker. Limited multi-cloud or hybrid cloud support. Cloud-native platform supporting all public clouds (AWS, Azure, GCP), Linux, Windows, Kubernetes, and on-premise environments. **Ease of Use** Easy to set up and use for its specific function. Requires Spinnaker and MySQL for full integration. User-friendly web interface and CLI. May require more initial setup and configuration than Chaos Monkey. **Cost** Free and open-source. Requires payment for advanced features and enterprise use. **Reporting & Analytics** No built-in detailed reporting or analysis; requires custom code for outage detection and fault tolerance. Offers rich analytics and visualization tools, automatic analysis, and storage of results. Integrates with Jira for action item tracking. **Safety & Risk Mitigation** May cause system downtime and false positives. High risk if unprepared due to randomness. Designed for safety and security. Allows starting small and scaling experiments, with features to confidently recreate incidents. **Additional Features** Basic functionality for instance termination. GameDay Manager, scenario sharing, scheduled scenarios, reliability scoring, dependency discovery, failure flags, private edition. Scan the table for the main differences and the trade-offs behind them. When choosing between these tools, consider your priorities. If cost is your main concern and you only need basic random instance termination, Chaos Monkey (or similar open-source tools like Pumba for Docker/Kubernetes) could be a good starting point. However, if you need precise control over fault injection, diverse fault types, full multi-cloud support, advanced features like GameDay management, and detailed analytics, Gremlin is the stronger choice for enterprise-level reliability efforts. Ultimately, your decision should factor in your team’s chaos engineering maturity, available budget, and the complexity of the systems you’re testing. The clear difference between Chaos Monkey and Gremlin reflects how Chaos Engineering has become a commercial and professional discipline. What started as an internal Netflix experiment has grown into a dedicated industry with advanced platforms. Reliability is now treated as a core business function. ## [How do we inject chaos in AWS environments?](#how-do-we-inject-chaos-in-aws-environments) In AWS environments, Chaos Engineering follows a structured framework designed to find resilience gaps in your workloads. It’s not about randomly breaking production systems. It’s a valuable tool for understanding how your workloads behave under simulated failure conditions. The most common approach uses the AWS Fault Injection Simulator (FIS), a managed service built specifically for running chaos engineering experiments on AWS services. Here’s the typical workflow: 1. **Define steady state:** First, define the normal operating condition for your systems. This baseline lets you measure what happens when you inject chaos. Collect and analyze data during stable conditions to set performance baselines and identify normal behavior patterns. 2. **Design chaos tests:** Plan controlled chaos experiments to simulate different failure scenarios within that steady state. Identify specific components or services to target and determine the experiment’s scope and severity. AWS FIS is perfect for this. 3. **Execute experiments:** Set up the necessary infrastructure for running tests, including test environments, monitoring, and logging systems. Then run your defined experiments. 4. **Analyze and fix:** During experiments, continuously monitor system behavior. Collect and analyze data to assess performance, stability, and resilience impacts, comparing against your baselines. 5. **Iterate and improve:** Repeat these steps periodically so your system stays resilient over time. Here’s a practical example using AWS FIS to terminate EC2 instances: aws-fis-ec2-termination.json ``` 1{2 "description": "Test application resilience by terminating EC2 instances",3 "targets": {4 "ec2-instances": {5 "resourceType": "aws:ec2:instance",6 "resourceTags": {7 "Environment": "staging",8 "ChaosReady": "true"9 },10 "selectionMode": "COUNT(2)"11 }12 },13 "actions": {14 "terminate-instances": {15 "actionId": "aws:ec2:terminate-instances",16 "parameters": {},17 "targets": {18 "Instances": "ec2-instances"19 }20 }21 },22 "stopConditions": [23 {24 "source": "aws:cloudwatch:alarm",25 "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:HighErrorRate"26 }27 ],28 "roleArn": "arn:aws:iam::123456789012:role/FISExperimentRole",29 "tags": {30 "Name": "EC2-Termination-Test",31 "Team": "Platform"32 }33} ``` To run this experiment: run-aws-fis-experiment.sh ``` #!/bin/bash # Create the experiment templateTEMPLATE_ID=$(aws fis create-experiment-template \ --cli-input-json file://aws-fis-ec2-termination.json \ --query 'experimentTemplate.id' \ --output text) echo "Created experiment template: $TEMPLATE_ID" # Start the experimentEXPERIMENT_ID=$(aws fis start-experiment \ --experiment-template-id "$TEMPLATE_ID" \ --query 'experiment.id' \ --output text) echo "Started experiment: $EXPERIMENT_ID" # Monitor experiment statusaws fis get-experiment \ --id "$EXPERIMENT_ID" \ --query 'experiment.state.status' \ --output text ``` **Examples of chaos experiments in AWS services:** - **Amazon Aurora:** Simulate network latency between Aurora instances, introduce failures in replica instances, or test increased load on read/write capacity. - **Amazon Kinesis:** Simulate higher data ingestion rates to test stream scaling. - **Amazon EC2:** Test Spot Instance interruptions to verify application handling of sudden terminations. - **Amazon DynamoDB:** Deny traffic to/from regional endpoints to test failover mechanisms. A unique challenge arises with serverless environments like AWS Lambda. You don’t control or access the underlying infrastructure, making traditional fault injection difficult. Here are two approaches: **1\. Using a library in Lambda code:** You can inject faults directly within the Lambda function using a library: lambda-with-chaos.mjs ``` 1import {DynamoDBClient, PutItemCommand} from '@aws-sdk/client-dynamodb';2import {S3Client, GetObjectCommand} from '@aws-sdk/client-s3';3 4// Initialize AWS SDK v3 clients5const dynamoClient = new DynamoDBClient({region: process.env.AWS_REGION});6const s3Client = new S3Client({region: process.env.AWS_REGION});7 8/**9 * Chaos injection library for Lambda functions10 * Supports error injection, latency injection, and custom failure scenarios11 */12class ChaosInjector {13 constructor(config = {}) {14 this.errorRate = parseFloat(config.errorRate) || 0;15 this.latencyMs = parseInt(config.latencyMs, 10) || 0;16 this.failureTypes = config.failureTypes || ['generic_error'];17 this.enabled = config.enabled !== false;18 }19 20 /**21 * Inject artificial latency22 */23 async injectLatency() {24 if (this.latencyMs > 0) {25 console.log(`[Chaos] Injecting ${this.latencyMs}ms latency`);26 await new Promise((resolve) => setTimeout(resolve, this.latencyMs));27 }28 }29 30 /**31 * Inject random errors based on error rate32 */33 async injectError() {34 if (Math.random() < this.errorRate) {35 const failureType =36 this.failureTypes[Math.floor(Math.random() * this.failureTypes.length)];37 38 console.error(`[Chaos] Injecting failure: ${failureType}`);39 40 switch (failureType) {41 case 'timeout_error':42 throw new Error('Chaos: Simulated timeout error');43 case 'throttle_error':44 const error = new Error('Chaos: Simulated throttling');45 error.code = 'ThrottlingException';46 throw error;47 case 'service_unavailable':48 const unavailableError = new Error('Chaos: Service unavailable');49 unavailableError.statusCode = 503;50 throw unavailableError;51 default:52 throw new Error('Chaos: Simulated generic failure');53 }54 }55 }56 57 /**58 * Execute chaos injection59 */60 async inject() {61 if (!this.enabled) {62 return;63 }64 65 await this.injectLatency();66 await this.injectError();67 }68}69 70// Initialize chaos injector from environment variables71const chaos = new ChaosInjector({72 errorRate: process.env.CHAOS_ERROR_RATE || 0,73 latencyMs: process.env.CHAOS_LATENCY_MS || 0,74 failureTypes: process.env.CHAOS_FAILURE_TYPES?.split(',') || [75 'generic_error',76 ],77 enabled: process.env.CHAOS_ENABLED !== 'false',78});79 80/**81 * Lambda handler with chaos engineering82 */83export const handler = async (event, context) => {84 console.log('Processing request:', {requestId: context.requestId});85 86 try {87 // Inject chaos before processing88 await chaos.inject();89 90 // Your actual business logic91 const result = await processRequest(event);92 93 return {94 statusCode: 200,95 headers: {96 'Content-Type': 'application/json',97 'X-Request-Id': context.requestId,98 },99 body: JSON.stringify(result),100 };101 } catch (error) {102 console.error('Error processing request:', {103 error: error.message,104 stack: error.stack,105 requestId: context.requestId,106 });107 108 return {109 statusCode: error.statusCode || 500,110 headers: {111 'Content-Type': 'application/json',112 'X-Request-Id': context.requestId,113 },114 body: JSON.stringify({115 error: error.message,116 requestId: context.requestId,117 }),118 };119 }120};121 122/**123 * Example business logic using AWS SDK v3124 */125async function processRequest(event) {126 const {userId, action} = JSON.parse(event.body || '{}');127 128 // Example: Write to DynamoDB using AWS SDK v3129 if (action === 'save') {130 const command = new PutItemCommand({131 TableName: process.env.TABLE_NAME,132 Item: {133 userId: {S: userId},134 timestamp: {N: Date.now().toString()},135 data: {S: JSON.stringify(event.body)},136 },137 });138 139 await dynamoClient.send(command);140 }141 142 // Example: Read from S3 using AWS SDK v3143 if (action === 'fetch') {144 const command = new GetObjectCommand({145 Bucket: process.env.BUCKET_NAME,146 Key: `users/${userId}/data.json`,147 });148 149 const response = await s3Client.send(command);150 const data = await response.Body.transformToString();151 return {data: JSON.parse(data)};152 }153 154 return {message: 'Success', userId, action};155} ``` **2\. Using a Lambda extension:** You can deploy a Lambda layer that injects failures without changing your main function code: chaos-extension/extension.py ``` 1#!/usr/bin/env python32"""Lambda Extension for Chaos Engineering3 4This extension intercepts Lambda invocations and injects controlled failures5to test system resilience without modifying application code.6"""7 8import os9import sys10import json11import random12import time13import signal14import requests15from pathlib import Path16from typing import Dict, List, Optional, Tuple17from datetime import datetime18 19# Lambda Extensions API endpoint20EXTENSION_API = f"http://{os.getenv('AWS_LAMBDA_RUNTIME_API')}/2020-01-01/extension"21 22 23class ChaosExtension:24 """Chaos injection engine for Lambda functions"""25 26 def __init__(self):27 self.extension_id: Optional[str] = None28 self.error_rate = float(os.getenv('CHAOS_ERROR_RATE', '0.0'))29 self.latency_ms = int(os.getenv('CHAOS_LATENCY_MS', '0'))30 self.max_latency_ms = int(os.getenv('CHAOS_MAX_LATENCY_MS', '5000'))31 self.enabled = os.getenv('CHAOS_ENABLED', 'true').lower() == 'true'32 33 try:34 self.failure_types = json.loads(35 os.getenv('CHAOS_FAILURE_TYPES', '["http_error"]')36 )37 except json.JSONDecodeError:38 self.failure_types = ['http_error']39 print('[chaos-extension] Warning: Invalid CHAOS_FAILURE_TYPES, using default')40 41 # Validate configuration42 self._validate_config()43 44 def _validate_config(self) -> None:45 """Validate chaos configuration parameters"""46 if not 0 <= self.error_rate <= 1:47 print(f'[chaos-extension] Warning: Invalid error_rate {self.error_rate}, clamping to [0,1]')48 self.error_rate = max(0, min(1, self.error_rate))49 50 if self.latency_ms < 0:51 print(f'[chaos-extension] Warning: Negative latency {self.latency_ms}ms, setting to 0')52 self.latency_ms = 053 54 if self.latency_ms > self.max_latency_ms:55 print(f'[chaos-extension] Warning: Latency {self.latency_ms}ms exceeds max {self.max_latency_ms}ms')56 self.latency_ms = self.max_latency_ms57 58 def register(self) -> str:59 """Register extension with Lambda Extensions API"""60 try:61 response = requests.post(62 f'{EXTENSION_API}/register',63 json={'events': ['INVOKE', 'SHUTDOWN']},64 headers={'Lambda-Extension-Name': 'chaos-extension'},65 timeout=566 )67 response.raise_for_status()68 self.extension_id = response.headers['Lambda-Extension-Identifier']69 print(f'[chaos-extension] Registered with ID: {self.extension_id}')70 return self.extension_id71 except Exception as e:72 print(f'[chaos-extension] Failed to register: {e}')73 sys.exit(1)74 75 def next_event(self) -> Dict:76 """Wait for next Lambda event"""77 try:78 response = requests.get(79 f'{EXTENSION_API}/event/next',80 headers={'Lambda-Extension-Identifier': self.extension_id},81 timeout=None82 )83 response.raise_for_status()84 return response.json()85 except Exception as e:86 print(f'[chaos-extension] Error getting next event: {e}')87 sys.exit(1)88 89 def should_inject_chaos(self) -> bool:90 """Determine if chaos should be injected for this invocation"""91 if not self.enabled:92 return False93 return random.random() < self.error_rate94 95 def inject_latency(self) -> None:96 """Inject artificial latency"""97 if self.latency_ms > 0:98 actual_latency = random.randint(99 self.latency_ms // 2,100 self.latency_ms101 )102 print(f'[chaos-extension] Injecting {actual_latency}ms latency')103 time.sleep(actual_latency / 1000.0)104 105 def inject_error(self) -> Optional[Tuple[int, Dict]]:106 """Inject simulated error based on failure type"""107 if not self.should_inject_chaos():108 return None109 110 failure_type = random.choice(self.failure_types)111 timestamp = datetime.utcnow().isoformat()112 113 print(f'[chaos-extension] Injecting failure: {failure_type} at {timestamp}')114 115 error_scenarios = {116 'http_error': (500, {117 'error': 'Chaos: Simulated HTTP 500 Internal Server Error',118 'type': 'InternalServerError',119 'timestamp': timestamp120 }),121 'timeout': (408, {122 'error': 'Chaos: Simulated request timeout',123 'type': 'RequestTimeout',124 'timestamp': timestamp125 }),126 'throttle': (429, {127 'error': 'Chaos: Simulated throttling',128 'type': 'ThrottlingException',129 'timestamp': timestamp130 }),131 'service_unavailable': (503, {132 'error': 'Chaos: Service temporarily unavailable',133 'type': 'ServiceUnavailable',134 'timestamp': timestamp135 }),136 'bad_gateway': (502, {137 'error': 'Chaos: Bad gateway response',138 'type': 'BadGateway',139 'timestamp': timestamp140 })141 }142 143 if failure_type == 'timeout':144 # Simulate timeout with actual delay145 timeout_duration = random.randint(1, 5)146 print(f'[chaos-extension] Simulating {timeout_duration}s timeout')147 time.sleep(timeout_duration)148 149 return error_scenarios.get(150 failure_type,151 (500, {'error': 'Chaos: Unknown failure type', 'timestamp': timestamp})152 )153 154 def process_invoke(self, event: Dict) -> None:155 """Process INVOKE event"""156 request_id = event.get('requestId', 'unknown')157 print(f'[chaos-extension] Processing invocation: {request_id}')158 159 # Inject latency before function execution160 self.inject_latency()161 162 # Check if error should be injected163 error = self.inject_error()164 if error:165 status_code, error_body = error166 print(f'[chaos-extension] Chaos injected: {status_code} - {error_body["error"]}')167 168 def run(self) -> None:169 """Main extension loop"""170 print('[chaos-extension] Starting chaos extension')171 print(f'[chaos-extension] Configuration:')172 print(f' - Enabled: {self.enabled}')173 print(f' - Error rate: {self.error_rate * 100:.1f}%')174 print(f' - Latency: {self.latency_ms}ms (max: {self.max_latency_ms}ms)')175 print(f' - Failure types: {self.failure_types}')176 177 # Register extension178 self.register()179 180 # Main event loop181 while True:182 event = self.next_event()183 event_type = event.get('eventType')184 185 if event_type == 'INVOKE':186 self.process_invoke(event)187 elif event_type == 'SHUTDOWN':188 print('[chaos-extension] Shutdown event received')189 break190 else:191 print(f'[chaos-extension] Unknown event type: {event_type}')192 193 194def signal_handler(signum, frame):195 """Handle shutdown signals gracefully"""196 print(f'[chaos-extension] Received signal {signum}, shutting down')197 sys.exit(0)198 199 200if __name__ == '__main__':201 # Register signal handlers202 signal.signal(signal.SIGTERM, signal_handler)203 signal.signal(signal.SIGINT, signal_handler)204 205 try:206 chaos = ChaosExtension()207 chaos.run()208 except Exception as e:209 print(f'[chaos-extension] Fatal error: {e}')210 sys.exit(1) ``` ## [References](#references) 1. What is Chaos Engineering? | OpenText, accessed on June 6, 2025, [https://www.opentext.com/what-is/chaos-engineering](https://www.opentext.com/what-is/chaos-engineering) 2. Chaos Engineering Tutorial: Comprehensive Guide With Best Practices - LambdaTest, accessed on June 6, 2025, [https://www.lambdatest.com/learning-hub/chaos-engineering-tutorial](https://www.lambdatest.com/learning-hub/chaos-engineering-tutorial) 3. Home - Chaos Monkey, accessed on June 6, 2025, [https://netflix.github.io/chaosmonkey/](https://netflix.github.io/chaosmonkey/) 4. Netflix/chaosmonkey: Chaos Monkey is a resiliency tool … - GitHub, accessed on June 6, 2025, [https://github.com/Netflix/chaosmonkey](https://github.com/Netflix/chaosmonkey) 5. Chaos Engineering | Gremlin, accessed on June 6, 2025, [https://www.gremlin.com/product/chaos-engineering](https://www.gremlin.com/product/chaos-engineering) 6. Comparing Chaos Engineering tools - Gremlin, accessed on June 6, 2025, [https://www.gremlin.com/community/tutorials/chaos-engineering-tools-comparison](https://www.gremlin.com/community/tutorials/chaos-engineering-tools-comparison) 7. Chaos Monkey vs Gremlin vs Pumba - Bwiza Charlotte, accessed on June 6, 2025, [https://bwiza.hashnode.dev/chaos-monkey-vs-gremlin-vs-pumba](https://bwiza.hashnode.dev/chaos-monkey-vs-gremlin-vs-pumba) 8. AWS chaos engineering tools: PwC, accessed on June 6, 2025, [https://www.pwc.com/us/en/technology/alliances/library/aws-chaos-engineering.html](https://www.pwc.com/us/en/technology/alliances/library/aws-chaos-engineering.html) 9. Introduction to Chaos Engineering in Serverless Architectures - Ran The Builder, accessed on June 6, 2025, [https://www.ranthebuilder.cloud/post/introduction-to-chaos-engineering-serverless](https://www.ranthebuilder.cloud/post/introduction-to-chaos-engineering-serverless) 10. information system resilience - Glossary - NIST CSRC, accessed on June 6, 2025, [https://csrc.nist.gov/glossary/term/information\_system\_resilience](https://csrc.nist.gov/glossary/term/information_system_resilience) 11. Chaos Engineering Explained: Core Principles and … - Distant Job, accessed on June 6, 2025, [https://distantjob.com/blog/chaos-engineering/](https://distantjob.com/blog/chaos-engineering/) 12. Types of chaos experiments - Fork My Brain, accessed on June 6, 2025, [https://notes.nicolevanderhoeven.com/Types+of+chaos+experiments](https://notes.nicolevanderhoeven.com/Types+of+chaos+experiments) 13. Gremlin vs Harness CE: Chaos Engineering Comparison, accessed on June 6, 2025, [https://www.harness.io/comparison-guide/gremlin-vs-harness](https://www.harness.io/comparison-guide/gremlin-vs-harness) 14. I’m Kolton Andrus, Ask Me Anything about Chaos Engineering - Atlassian Community, accessed on June 6, 2025, [https://community.atlassian.com/forums/Jira-questions/I-m-Kolton-Andrus-Ask-Me-Anything-about-Chaos-Engineering/qaq-p/1342117](https://community.atlassian.com/forums/Jira-questions/I-m-Kolton-Andrus-Ask-Me-Anything-about-Chaos-Engineering/qaq-p/1342117) 15. Chaos Engineering with Chaos Mesh and vCluster: Testing Close to …, accessed on June 6, 2025, [https://www.loft.sh/blog/chaos-mesh-with-vcluster](https://www.loft.sh/blog/chaos-mesh-with-vcluster) Was this useful? ## Tags [#Chaos Engineering](/blog/tags/chaos-engineering)[#Chaos Monkey](/blog/tags/chaos-monkey)[#Gremlin](/blog/tags/gremlin)[#System Resiliency](/blog/tags/system-resiliency)[#Fault Injection](/blog/tags/fault-injection)[#AWS](/blog/tags/aws)[#Kubernetes](/blog/tags/kubernetes)[#Production Testing](/blog/tags/production-testing)[#Site Reliability Engineering](/blog/tags/site-reliability-engineering)[#SRE](/blog/tags/sre)[#Netflix](/blog/tags/netflix)[#System Testing](/blog/tags/system-testing)[#Reliability Testing](/blog/tags/reliability-testing) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin&url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin&title=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin&summary=How%20Chaos%20Engineering%20builds%20resilient%20systems.%20Chaos%20Monkey%20and%20Gremlin%20inject%20faults%20in%20AWS%20and%20Kubernetes%20so%20you%20find%20the%20weak%20spots%20before%20an%20outage%20does.%20Best%20practices%2C%20experiment%20types%2C%20and%20frequently%20asked%20questions%20for%20resilient%20software%20testing.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin%20https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin&text=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin&title=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin&t=Chaos%20Engineering%3A%20Testing%20Resiliency%20with%20Chaos%20Monkey%20and%20Gremlin "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fchaos-engineering-resiliency-testing-monkey-gremlin&media=&description=How%20Chaos%20Engineering%20builds%20resilient%20systems.%20Chaos%20Monkey%20and%20Gremlin%20inject%20faults%20in%20AWS%20and%20Kubernetes%20so%20you%20find%20the%20weak%20spots%20before%20an%20outage%20does.%20Best%20practices%2C%20experiment%20types%2C%20and%20frequently%20asked%20questions%20for%20resilient%20software%20testing. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Designing SLOs and Error Budgets: Your Blueprint for Sustainable Reliability](/_astro/hero.C1145KtV_oyFc5.webp)](/blog/post/designing-slos-error-budgets-reliability-blueprint) ## [Designing SLOs and Error Budgets: Your Blueprint for Sustainable Reliability](/blog/post/designing-slos-error-budgets-reliability-blueprint) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [SRE](/blog/categories/sre) - [SLO](/blog/categories/slo) - [Error Budgets](/blog/categories/error-budgets) - [Reliability](/blog/categories/reliability) Every business shipping software is trying to balance two big things: getting new features out fast and keeping their services super reliable. Every tech team deals with this, pushing for new ideas wh [#SLI](/blog/tags/sli)[#SLA](/blog/tags/sla)[#Monitoring](/blog/tags/monitoring)+2 tags [read more](/blog/post/designing-slos-error-budgets-reliability-blueprint) [![Testing Terraform: Static Analysis, Native Tests, and Terratest](/_astro/hero.QFNhZZd9_Z1MKEy5.webp)](/blog/post/terraform-testing-terratest-native-tests) ## [Testing Terraform: Static Analysis, Native Tests, and Terratest](/blog/post/terraform-testing-terratest-native-tests) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Infrastructure as Code](/blog/categories/infrastructure-as-code) - [DevOps](/blog/categories/devops) - [Testing](/blog/categories/testing) If you treat infrastructure as code, you have to test it like code. Most of us have lived the alternative. You change one input on a shared module, run a quick plan against staging, and merge. A few h [#Terraform](/blog/tags/terraform)[#Terratest](/blog/tags/terratest)[#CI/CD](/blog/tags/cicd)+4 tags [read more](/blog/post/terraform-testing-terratest-native-tests) [![Building Resilient Systems: Immutable Infrastructure with Packer and Terraform](/_astro/hero.C1--9UB8_ZaNEHL.webp)](/blog/post/immutable-infrastructure-packer-terraform-guide) ## [Building Resilient Systems: Immutable Infrastructure with Packer and Terraform](/blog/post/immutable-infrastructure-packer-terraform-guide) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps](/blog/categories/devops) - [Infrastructure as Code](/blog/categories/infrastructure-as-code) - [Cloud Computing](/blog/categories/cloud-computing) What is immutable infrastructure? The way we manage IT infrastructure has really changed. We're moving from old-school, changeable setups to more modern, "immutable" ones. Understanding this big s [#Packer](/blog/tags/packer)[#Terraform](/blog/tags/terraform)[#Immutable Infrastructure](/blog/tags/immutable-infrastructure)+3 tags [read more](/blog/post/immutable-infrastructure-packer-terraform-guide) [![Taming the Chaos: Let's Sort Out Those Flaky CI/CD Pipelines](/_astro/hero.CNGGank__Zjy3fe.webp)](/blog/post/troubleshooting-flaky-ci-cd-pipelines) ## [Taming the Chaos: Let's Sort Out Those Flaky CI/CD Pipelines](/blog/post/troubleshooting-flaky-ci-cd-pipelines) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [CI/CD](/blog/categories/cicd) - [Testing](/blog/categories/testing) - [DevOps](/blog/categories/devops) - [Automation](/blog/categories/automation) - [Pipeline Reliability](/blog/categories/pipeline-reliability) Ever get super frustrated with your CI/CD pipeline? You know, the one that sometimes works perfectly and other times just throws a random tantrum? You push your code, the pipeline starts doing its thi [#Flaky Tests](/blog/tags/flaky-tests)[#CI/CD Pipelines](/blog/tags/cicd-pipelines)[#Test Automation](/blog/tags/test-automation)+5 tags [read more](/blog/post/troubleshooting-flaky-ci-cd-pipelines) [![Modular Terraform for Scalable Infrastructure as Code](/_astro/hero.kBsnpbcJ_1mG2Sh.webp)](/blog/post/modular-terraform-scalable-iac-guide) ## [Modular Terraform for Scalable Infrastructure as Code](/blog/post/modular-terraform-scalable-iac-guide) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Infrastructure as Code](/blog/categories/infrastructure-as-code) - [Terraform](/blog/categories/terraform) - [DevOps](/blog/categories/devops) - [Cloud Engineering](/blog/categories/cloud-engineering) - [Automation](/blog/categories/automation) Businesses need infrastructure that's flexible and can grow fast, and managing it by hand doesn't scale. Infrastructure as Code, or IaC, changed how we build and manage those digital foundations. IaC [#Terraform](/blog/tags/terraform)[#Infrastructure as Code](/blog/tags/infrastructure-as-code)[#IaC](/blog/tags/iac)+12 tags [read more](/blog/post/modular-terraform-scalable-iac-guide) [![Navigating Growth: Building a Secure and Scalable AWS Environment with a Multi-Account Architecture and Control Tower](/_astro/hero.BShn8B1r_Zu8h2W.webp)](/blog/post/multi-account-aws-control-tower) ## [Navigating Growth: Building a Secure and Scalable AWS Environment with a Multi-Account Architecture and Control Tower](/blog/post/multi-account-aws-control-tower) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud Computing](/blog/categories/cloud-computing) - [AWS](/blog/categories/aws) - [DevOps](/blog/categories/devops) - [Cloud Security](/blog/categories/cloud-security) - [Cloud Architecture](/blog/categories/cloud-architecture) The cloud journey often kicks off with a single AWS account. It feels simple and straightforward, especially when you're just starting out or have smaller teams. But as your cloud usage grows, that in [#AWS](/blog/tags/aws)[#Control Tower](/blog/tags/control-tower)[#Multi Account Strategy](/blog/tags/multi-account-strategy)+8 tags [read more](/blog/post/multi-account-aws-control-tower) 6 related posts