AWS Kafka / MSK Engineer EventBridge & Schema Registry

Key Responsibilities

AWS Glue Schema Registry

• Manage and maintain AWS Glue Schema Registry for event schema governance.
• Validate and verify producer/consumer schemas to ensure compatibility compliance.
• Implement schema versioning strategies and compatibility checks.
• Troubleshoot schema evolution and serialization/deserialization issues.

Amazon EventBridge

• Configure and manage EventBridge event buses, rules, and targets.
• Design and implement event filtering patterns for business applications.
• Monitor event routing and troubleshoot event delivery failures.
• Integrate AWS services using EventBridge for event-driven workflows.
• Optimize event filtering rules to improve efficiency and reduce unnecessary processing.

Amazon MSK (Managed Streaming for Apache Kafka)

• Administer and support Amazon MSK clusters.
• Monitor Kafka brokers, topics, partitions, and consumer groups.
• Troubleshoot producer and consumer connectivity issues.
• Implement Kafka security controls including IAM/SASL authentication and encryption.
• Support high availability, scaling, and performance tuning activities.
• Monitor cluster health using CloudWatch and Kafka metrics.

Cloud Operations & Support

• Perform L2 production support and incident management.
• Analyze logs and metrics using CloudWatch and related monitoring tools.
• Participate in root cause analysis (RCA) and problem management.
• Implement operational automation using AWS services and scripting.
• Collaborate with development and architecture teams for platform improvements.

Required Technical Skills

AWS Services

• AWS Glue Schema Registry
• Amazon EventBridge
• Amazon MSK (Kafka)
• AWS Lambda
• AWS IAM
• AWS CloudWatch
• AWS Secrets Manager
• AWS KMS
• Amazon S3

Messaging & Streaming

• Apache Kafka concepts
• Kafka Connect
• Consumer Groups
• Event-Driven Architecture
• Schema Registry Management

Development & Automation

• Python
• Shell scripting
• Infrastructure as Code (Terraform or CloudFormation)
• Git/GitHub

Monitoring & Troubleshooting

• CloudWatch Logs & Metrics
• Kafka Monitoring Tools
• Performance Analysis
• Incident Management