AWS Cloud Cost Optimization: Stale EBS Snapshot Cleanup

Project Overview

This project implements an automated, serverless solution for identifying and removing orphaned EBS snapshots across an AWS account. The Lambda-based automation targets a common cost optimization challenge: EBS snapshots persisting after their associated volumes have been deleted or detached, generating unnecessary storage charges.

The solution follows infrastructure-as-code principles and least-privilege IAM security postures, making it suitable for production environments with stringent compliance requirements.

---

Problem Statement

The Challenge

EBS snapshots incur storage costs regardless of whether their source volumes remain active or have been deleted. In dynamic environments with frequent instance and volume provisioning, orphaned snapshots accumulate silently:

• Volume Deletion Scenarios: When an EC2 instance is terminated, attached volumes may be deleted automatically (depending on DeleteOnTermination settings), but snapshots created from those volumes persist indefinitely.
• Detached Volumes: Volumes explicitly detached from instances but not deleted create orphaned snapshots with no clear lineage to active resources.
• Manual Snapshot Creation: Snapshots created for backup purposes may no longer map to any running infrastructure.
• Cost Impact: At $0.05 per GB-month (EBS Snapshot Storage pricing), a single 500 GB snapshot generates $25/month in recurring costs. Enterprise accounts with thousands of snapshots can accumulate $5,000–$50,000+ in annual unnecessary snapshot storage costs.

Root Cause

EBS snapshots and their source volumes operate as independent AWS resources. Deleting a volume does not automatically clean up associated snapshots, creating an orphan management problem particularly acute in:

• Multi-team environments with decentralized resource governance
• Development/testing accounts with high infrastructure churn
• Accounts leveraging Infrastructure as Code (IaC) that periodically redeploys entire stacks

---

Solution Overview

This project automates snapshot orphan detection and remediation through a Python-based AWS Lambda function that:

1. Enumerates all EBS snapshots owned by the AWS account
2. Validates snapshot-to-volume lineage against active EC2 infrastructure
3. Identifies orphaned snapshots whose source volumes no longer exist or are not attached to running instances
4. Removes orphaned snapshots to eliminate unnecessary storage costs
5. Logs all operations to CloudWatch Logs for audit trails and cost tracking

The function operates entirely serverless, requires no EC2 infrastructure, and can be triggered on a configurable schedule via EventBridge.

---

High-Level Architecture

Components

AWS Lambda (Function Runtime)

• Executes Python code using boto3 SDK
• Triggered on schedule via EventBridge (recommended: daily or weekly)
• Assumes a custom IAM execution role with least-privilege permissions
• Logs all snapshot operations (enumeration, validation, deletion) to CloudWatch

Amazon EC2 & EBS

• EC2 instances serve as source-of-truth for active infrastructure
• EBS volumes represent actual storage attached to instances
• EBS snapshots are queried for orphan status

IAM Execution Role

• Custom role assumed by Lambda with fine-grained permissions
• Policies restrict actions to: DescribeSnapshots, DescribeInstances, DescribeVolumes, DeleteSnapshot, CloudWatch Logs
• No wildcard () permissions; all resources explicitly scoped to ec2: namespace

CloudWatch Logs

• Captures all Lambda execution logs
• Tracks snapshot enumeration counts, validation results, and deletion metrics
• Enables cost tracking and operational monitoring

EventBridge (Optional)

• Schedules Lambda function on configurable cadence (e.g., daily at 2:00 AM UTC)
• Decouples scheduling from function logic

Data Flow

┌─────────────────────────────────────────────────────────────────┐
│ EventBridge Trigger (Scheduled)                                  │
└─────────────┬───────────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Lambda Function (Python + boto3)                                 │
│                                                                  │
│ ┌──────────────────────────────────────────────────────────┐   │
│ │ 1. Create EC2 Client                                    │   │
│ │ 2. Query All Snapshots (OwnerIds=['self'])             │   │
│ │ 3. Fetch Running Instances & Volumes                   │   │
│ │ 4. For Each Snapshot:                                  │   │
│ │    - Check if volume still exists                      │   │
│ │    - Check if volume is attached to running instance   │   │
│ │    - Mark as orphaned if conditions not met            │   │
│ │ 5. Delete Orphaned Snapshots                           │   │
│ │ 6. Log Results to CloudWatch                           │   │
│ └──────────────────────────────────────────────────────────┘   │
└──────────────────┬───────────────────────────────────────────────┘
                   │
         ┌─────────┴──────────┐
         │                    │
         ▼                    ▼
    AWS EC2 API         CloudWatch Logs
    (Queries &           (Audit Trail &
     Deletions)          Metrics)

---

Execution Flow

Lambda Function Logic

Step 1: Initialize AWS Clients

ec2_client = boto3.client('ec2')

Step 2: Query All Snapshots

The function retrieves all snapshots owned by the AWS account:

response = ec2_client.describe_snapshots(OwnerIds=['self'])
snapshots = response['Snapshots']

This returns snapshots regardless of state (completed, pending) or snapshot status. The OwnerIds=['self'] filter ensures only account-owned snapshots are processed, excluding shared snapshots.

Step 3: Build Active Volume Inventory

The function fetches all active EC2 instances and their attached volumes:

instances = ec2_client.describe_instances(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
)
active_volumes = set()
for reservation in instances['Reservations']:
    for instance in reservation['Instances']:
        for volume in instance['BlockDeviceMappings']:
            active_volumes.add(volume['Ebs']['VolumeId'])

This creates a set of volume IDs actively attached to running instances, serving as the baseline for orphan detection.

Step 4: Validate Each Snapshot

For each snapshot, the function:

4a. Check if VolumeId exists in Active Volumes

if snapshot['VolumeId'] not in active_volumes:
    # Snapshot is orphaned (volume deleted or detached)

4b. Validate Volume Still Exists The function makes an additional API call to confirm the volume still exists in the account:

try:
    ec2_client.describe_volumes(VolumeIds=[snapshot['VolumeId']])
except ec2_client.exceptions.ClientError as e:
    if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
        # Volume has been deleted; snapshot is orphaned

This accounts for race conditions where a volume may exist but no running instance claims it.

Step 5: Delete Orphaned Snapshots

Snapshots failing validation are deleted:

ec2_client.delete_snapshot(SnapshotId=snapshot['SnapshotId'])

Step 6: Log and Report

All operations are logged to CloudWatch with counts and details:

• Total snapshots enumerated
• Snapshots skipped (already attached to active instances)
• Orphaned snapshots identified
• Successfully deleted snapshots
• Failed deletions (with error details)

Key Decision Points

Condition	Decision	Rationale
Snapshot's VolumeId not in active volumes	Orphaned	Volume deleted or detached; no active lineage
Volume exists but not attached to any instance	Orphaned	Disconnected from infrastructure; eligible for cleanup
Snapshot in pending state	Skip	Incomplete snapshots may complete and become valid
Volume attached to stopped instance	Skip (Configurable)	Instance may restart; preserve snapshot
Encrypted snapshot with no volume	Orphaned	Cannot re-attach; represents unrecoverable cost

---

AWS Services & Technologies

Service	Purpose	Interaction
AWS Lambda	Serverless compute runtime	Executes Python function on schedule
Amazon EC2	Instance metadata and state	Source-of-truth for active infrastructure
Amazon EBS	Snapshot storage and lifecycle	Target for snapshot deletion queries
IAM	Access control and authentication	Execution role with least-privilege permissions
CloudWatch Logs	Operational logging	Audit trail and cost metrics
EventBridge	Scheduling and triggering	Orchestrates periodic Lambda invocations
boto3	AWS SDK (Python)	Programmatic API interactions

---

IAM & Security Model

Execution Role Design

A custom IAM role is created with the following least-privilege policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DescribeSnapshotsAndVolumes",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSnapshots",
        "ec2:DescribeInstances",
        "ec2:DescribeVolumes"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DeleteOrphanedSnapshots",
      "Effect": "Allow",
      "Action": ["ec2:DeleteSnapshot"],
      "Resource": "arn:aws:ec2:*:ACCOUNT_ID:snapshot/*"
    },
    {
      "Sid": "CloudWatchLogging",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:ACCOUNT_ID:log-group:/aws/lambda/*"
    }
  ]
}

Security Rationale

• Describe-only permissions on snapshots/instances: Read-only access required to enumerate resources and identify orphans.
• Resource-scoped DeleteSnapshot: Deletion permission scoped to snapshot resource ARN only (no instance/volume deletion).
• CloudWatch Logs scoped to Lambda: Logging limited to Lambda log groups, preventing cross-service log access.
• No wildcard actions: Each permission explicitly named (DescribeSnapshots, not Describe*).
• Account-scoped resources: All ARNs include ACCOUNT_ID to prevent cross-account access.

Trust Relationship

The role must trust the Lambda service:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

---

Cost Optimization Impact

Cost Model

EBS Snapshot Storage Cost: $0.05 per GB per month (as of 2026)

ROI Calculation Example

Assume an AWS account with 100 GB of orphaned snapshots accumulating monthly due to infrastructure churn:

Metric	Value
Cumulative Orphaned Snapshots (monthly)	100 GB
Annual Accumulation	1,200 GB (1.2 TB)
Annual Storage Cost	$600 USD
Cleanup Cost (Lambda + API calls)	< $1 USD/month
Net Annual Savings	~$600 USD

For larger enterprises:

• 1 TB orphaned snapshots: $50/month × 12 = $600/year savings
• 10 TB orphaned snapshots: $500/month × 12 = $6,000/year savings
• 50 TB orphaned snapshots: $2,500/month × 12 = $30,000/year savings

Implementation Costs

• Lambda execution time: ~2–10 seconds per run (depending on snapshot count), consuming ~100–500 MB memory
• API call costs: Negligible; describe operations are free; delete operations incur standard EC2 API charges (minimal)
• CloudWatch Logs: ~1 KB per execution, ~30 GB/year at current pricing

Total annual implementation cost: < $50 USD

---

Monitoring & Logging

CloudWatch Log Structure

The Lambda function outputs st