Summary: The role combines Kafka platform engineering, security and network engineering, and senior Kafka development. The engineer will install, configure, secure, automate, monitor, and validate enterprise Apache Kafka environments while developing and supporting reliable event-driven applications, integrations, producers, consumers, stream-processing components, and operational tooling.
Key Responsibilities
Install, configure, upgrade, patch, and operate Apache Kafka clusters using KRaft mode across Dev, QA, UAT, Production, and disaster-recovery environments.
Configure brokers, controllers, topics, partitions, replication factors, retention policies, quotas, consumer groups, and MirrorMaker 2 replication.
Implement TLS/mTLS, SASL/SCRAM, Kafka ACLs, least-privilege access, certificate lifecycle management, secrets integration, audit logging, and security hardening.
Coordinate DNS, routing, firewall, load balancer, proxy, port, and network-segmentation requirements; troubleshoot connectivity, latency, certificate, and authentication issues.
Develop and support Java/Spring Boot Kafka producers and consumers, Kafka Streams applications, Kafka Connect integrations, serialization/deserialization, retry handling, dead-letter topics, idempotency, and error recovery.
Implement schema governance using Avro, JSON Schema, or Protobuf and support compatible schema evolution through an approved schema registry.
Build deployment and configuration automation using Ansible, Terraform, shell scripting, Git, and CI/CD pipelines.
Configure dashboards, logging, metrics, and alerts for broker health, under-replicated partitions, consumer lag, replication lag, storage, throughput, availability, and security events.
Perform functional, performance, failover, recovery, and security validation; diagnose production incidents and complete root-cause analysis.
Produce design documents, configuration standards, build guides, operating procedures, runbooks, test evidence, and knowledge-transfer materials; mentor developers and review code and configurations.
Required Skills & Experience
Strong hands-on experience with Apache Kafka architecture, KRaft, producers/consumers, consumer groups, partitioning, replication, offsets, delivery semantics, and cluster troubleshooting.
Proficiency in Java, Spring Boot, REST APIs, distributed systems, and event-driven design; working knowledge of Kafka Streams and Kafka Connect.
Practical experience with Linux/RHEL administration, TCP/IP, DNS, routing, firewalls, PKI, TLS certificates, keystores/truststores, authentication, and authorization.
Experience with monitoring and logging platforms such as Prometheus, Grafana, Splunk, or equivalent enterprise tools.
Experience with Git, CI/CD, infrastructure/configuration automation, performance tuning, DR testing, and production support.
Strong analytical, documentation, collaboration, communication, and mentoring skills.
Preferred: Knowledge of Confluent Kafka, including Confluent Platform or Confluent Cloud, Schema Registry, Kafka Connect, Control Center, ksqlDB, RBAC, Cluster Linking, Confluent CLI, managed connectors, and related monitoring and security capabilities, will be an added advantage. Confluent certification is desirable but not mandatory.
