Loading repository data…
Loading repository data…
aws-samples / repository
This project provides and example of end to end data processing application created using the combination of Amazon Managed Streaming for Apache Kafka (Amazon MSK), AWS Fargate, AWS Lambda and Amazon DynamoDB. Business logic is implemented in Java and Typescript. The build and deployment of the application is fully automated using AWS CDK.
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
This project provides an example of Apache Kafka data processing application. The build and deployment of the application if fully automated using AWS CDK.
Project consists of three main parts:
Provided example show cases two ways of packaging and deploying business logic using high-level AWS CDK constructs.
One way using Dockerfile and AWS CDK ECS ContainerImage to deploy Java application to AWS Fargate
and the other way is using AWS CDK NodejsFunction construct to deploy Typescript AWS Lambda code
Triggering the TransactionHandler Lambda function publishes messages to an Apache Kafka topic. The application is packaged in a container and deployed to ECS Fargate, consumes messages from the Kafka topic, processes them, and stores the results in an Amazon DynamoDB table. The KafkaTopicHandler Lambda function is called once during deployment to create Kafka topic. Both the Lambda function and the consumer application publish logs to Amazon CloudWatch.
amazon-msk-java-app-cdk/lib - directory containing all AWS CDK stacksamazon-msk-java-app-cdk/bin - directory containing AWS CDK app definitionamazon-msk-java-app-cdk/lambda - directory containing code of TransactionHandler AWS Lambda function as well as code of Custom Resource handler responsible for creating Kafka topic during deploymentconsumer - directory containing code of Kafka consumer Spring Boot Java applicationconsumer/docker/Dockerfile - definition of Docker image used for AWS Fargate container deploymentdoc - directory containing architecture diagramsscripts - directory containing deployment scriptsexport AWS_PROFILE=<REPLACE WITH YOUR AWS PROFILE NAME> or alternatively follow instructions in the AWS CDK documentationcd scriptsdeploy.sh scriptaccountId and value fields in the payload JSON and send multiple messages with different payloads to experiment with the application.
aws lambda invoke --cli-binary-format raw-in-base64-out --function-name TransactionHandler --log-type Tail --payload '{ "accountId": "account_123", "value": 456}' /dev/stdout --query 'LogResult' --output text | base64 -d
Accounts table.
aws dynamodb scan --table-name Accounts --query "Items[*].[id.S,Balance.N]" --output text
aws logs tail command with specified CloudWatch Logs groupFollow AWS CDK instructions to remove AWS CDK stacks from your account. You can also use scripts/destroy.sh.
Be sure to also manually remove Amazon DynamoDB table, clean up Amazon CloudWatch logs and remove Amazon ECR images to avoid incurring additional AWS infrastructure costs.