Senior Kafka Data Engineer

Aaura Softwares is seeking a Senior Kafka Data Engineer to lead the architecture and development of enterprise-scale streaming data pipelines. This role involves deep expertise in Kafka and requires collaboration with onsite US clients to translate business requirements into technical solutions and ensure high-quality data pipelines.

Responsibilities:

• Architect, build, and operate real‑time and near‑real‑time Kafka‑based pipelines
• Lead development using Confluent Kafka (core focus)
• Design and implement stream processing using:
• Apache Flink (preferred) or Apache Spark Structured Streaming
• Define Kafka best practices for:
• Topic design, partitioning, and delivery semantics
• Error handling and retries
• Backpressure management and performance tuning
• Design and maintain batch and hybrid ETL pipelines using SQL Server
• ETL tools preferred: SSIS
• Own data quality, reliability, and observability across streaming pipelines
• Work directly with onsite clients to:
• Translate business requirements into technical solutions
• Defend architectural decisions
• Troubleshoot complex Kafka and streaming production issues
• Mentor engineers and raise overall engineering standards
• Collaborate with DevOps and platform teams in containerized, distributed environments

Requirements:

• 15+ years of experience
• 8+ years of hands-on data engineering experience
• Deep expertise in Confluent Kafka
• Architecture and cluster design
• Topic and partition strategy
• Delivery semantics (exactly-once, at-least-once, etc.)
• Strong hands-on experience with Apache Flink or Apache Spark Structured Streaming
• Excellent SQL skills (complex analytics, performance tuning)
• Strong experience in Python, Java, or .NET (C#)
• Solid understanding of streaming and event-driven architectures
• ETL / ELT patterns
• Data modeling and orchestration
• Experience with SQL Server
• Apache Flink
• SSIS
• Docker / Kubernetes
• CI/CD for data pipelines
• Cloud or hybrid data platforms