Real-world AWS incident response and troubleshooting scenarios

🎯 TL;DR

Who I Am: 40-year-old delivery driver teaching myself cloud engineering to provide for my family (wife + 3 kids)

What This Is: Portfolio of intentionally broken AWS environments that I fix, document, and automate—proving I can do cloud support work, not just talk about it

Why It Matters:

• ✅ 4+ real incidents investigated with full RCA (not tutorials)
• ✅ 15+ Python/boto3 scripts automating AWS troubleshooting
• ✅ 10+ production-standard runbooks documenting fixes
• ✅ All from my AWS account (722631436033) - real work, real screenshots

What I'm Looking For: Entry-level cloud support/SysOps roles ($50k-$65k) where I can start immediately and prove myself through results

Investment: 50+ hours building this after 10-hour delivery shifts, $15/month from my paycheck running real AWS infrastructure

Bottom Line: I can't fake experience, so I'm building proof instead.

📑 Table of Contents

• 🎯 TL;DR
• 🎯 Project Purpose
• 🛠️ Technical Skills Demonstrated
• 📁 Repository Structure
• 🚀 Quick Start
• 📸 Lab Screenshots
• 📋 Available Scenarios
• 🔍 Learning Approach
• 📚 Runbook Examples
• 🎓 Why This Approach Works
• 🏗️ Built With
• 📊 Lab Metrics
• 🙋‍♂️ About Me
• 📧 Contact

This repository demonstrates hands-on cloud support skills through intentionally broken AWS environments. Each scenario simulates actual production issues requiring log analysis, metrics interpretation, root cause identification, and remediation—mirroring the daily work of cloud support engineers and SREs.

🎯 Project Purpose

Unlike guided tutorials that walk you through perfect deployments, this lab focuses on what happens when things break. Every scenario is designed to develop real troubleshooting muscle memory:

• Reading CloudWatch logs to identify failures
• Analyzing IAM permission errors
• Debugging Lambda function issues
• Investigating S3 bucket policy problems
• Troubleshooting VPC networking and security groups
• Documenting incidents and writing runbooks

For hiring managers: This project showcases practical cloud support capabilities—problem diagnosis, systematic troubleshooting, documentation, and automation—skills needed on day one of any cloud operations role.

🛠️ Technical Skills Demonstrated

AWS Services:

• EC2 (instances, security groups, networking)
• Lambda (functions, triggers, permissions)
• S3 (buckets, policies, versioning)
• IAM (roles, policies, least privilege)
• VPC (subnets, route tables, NACLs)
• CloudWatch (logs, metrics, alarms)

Tools & Languages:

• Python (boto3, AWS automation)
• PowerShell (Windows-based AWS management)
• Bash scripting (Linux troubleshooting)
• AWS CLI (resource inspection and remediation)
• Git/GitHub (version control, CI/CD basics)

Cloud Support Methodology:

• Incident response workflows
• Root cause analysis (5 Whys, fishbone)
• Runbook documentation
• Post-incident reviews
• Preventive controls

📁 Repository Structure

AWS_Error_Driven_Troubleshooting_Lab/
├── incidents/           # Real-world incident scenarios with full context
├── errors/              # Error messages, logs, and diagnostic data
├── runbooks/            # Step-by-step troubleshooting procedures
├── scripts/             # Python automation and remediation scripts
├── lambdas/             # Lambda function code for scenarios
├── tests/               # Validation scripts to verify fixes
├── diagrams/            # Architecture diagrams and visual aids
├── screenshots/         # Evidence of troubleshooting process
└── .github/workflows/   # CI/CD automation for testing scenarios

🚀 Quick Start

Prerequisites

• AWS Account (Free Tier eligible)
• AWS CLI configured with credentials
• Python 3.8+ with boto3
• Basic familiarity with AWS console

Setup

1. Clone the repository

   git clone https://github.com/charles-bucher/AWS_Error_Driven_Troubleshooting_Lab.git
   cd AWS_Error_Driven_Troubleshooting_Lab
   

2. Install dependencies

   pip install -r requirements.txt
   

3. Configure AWS credentials

   aws configure
   # Enter your AWS Access Key ID, Secret Access Key, and preferred region
   

4. Verify AWS access

   aws sts get-caller-identity
   

Running a Scenario

Each incident folder contains:

• scenario.md - The problem description and symptoms
• deploy.py or deploy.sh - Script to create the broken environment
• expected_errors.txt - What you should see when things fail
• solution.md - Root cause and remediation (check after troubleshooting!)

Example workflow:

# Deploy a broken scenario
cd incidents/01_lambda_s3_permission_denied
python deploy.py

# Investigate the issue using AWS CLI, console, and CloudWatch logs
# Document your findings

# Compare your solution with the provided runbook
cat ../../runbooks/01_lambda_s3_troubleshooting.md

# Clean up resources
python cleanup.py

📸 Lab Screenshots

Deployment & Setup:

AWS CLI Identity Check

AWS CLI configured and ready - verifying account access

S3 Buckets Created

S3 buckets deployed for troubleshooting scenarios

Lambda Functions

Lambda functions deployed - intentionally misconfigured for practice

Troubleshooting in Action:

Lab Fix Execution

PowerShell script executing remediation steps

Deploy Output

Initial deployment showing expected errors

Note: All screenshots are from my actual AWS account (Account ID: 722631436033, Region: us-east-1). No stock images or tutorial screenshots - just real troubleshooting work.

📋 Available Scenarios

Each scenario is production-realistic and requires systematic troubleshooting:

Coming Soon:

• EC2 Instance Store Data Loss
• RDS Connection Pool Exhaustion
• API Gateway 502 Bad Gateway
• CloudFront Cache Invalidation Issues

💡 Tip: Start with the "Beginner" scenarios if you're new to AWS troubleshooting. Each incident includes a scenario.md file explaining the problem and an expected_errors.txt showing what you should see.

🔍 Learning Approach

This lab follows an error-driven learning model:

1. Deploy - Run a script to create a broken AWS environment
2. Observe - Encounter real error messages and failures
3. Investigate - Use AWS tools to gather diagnostic information
4. Diagnose - Apply root cause analysis techniques
5. Remediate - Fix the issue using AWS CLI or console
6. Document - Write up your findings and prevention steps
7. Validate - Run tests to confirm the fix works

This mirrors actual cloud support work where you receive a ticket, investigate logs, identify root cause, implement a fix, and document the incident.

💡 Example Workflow:

# 1. Navigate to a scenario
cd incidents/01_lambda_s3_permission_denied/

# 2. Read the scenario description
cat scenario.md

# 3. Deploy the broken environment
python deploy.py
# Output: Resources deployed. Lambda will fail with permission errors.

# 4. Investigate using AWS CLI
aws lambda get-function --function-name troubleshooting-lambda
aws logs tail /aws/lambda/troubleshooting-lambda --follow

# 5. Identify the root cause
# (IAM role missing S3 permissions)

# 6. Implement the fix
aws iam attach-role-policy \
  --role-name lambda-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# 7. Validate the fix
python test_fix.py
# Output: ✅ Lambda executed successfully

# 8. Document your findings
# Compare with: cat ../../runbooks/01_lambda_s3_troubleshooting.md

# 9. Clean up resources
python cleanup.py

💰 Lab Costs

Estimated AWS costs per scenario: