Kafka Admin/Engineer
New York, NY (Hybrid)
Exp: 7+ Years
Need Only Locals who can attend In-Person Interview.
Job Summary:
The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.
Responsibilities
Design, implement, and maintain highly available Kafka clusters in production environments.
Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
Administer security protocols and access controls to protect the platform.
Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
Assist with planning and managing cluster resources to support future growth.
Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
Manage a tune of Kafka Connect, Kafka Streams, and Mirror Maker as needed for data pipelines.
Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).
Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
Perform root cause analysis and resolution of platform-level incidents and outages.
Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.
Qualifications/ Skills
Working experience with OpenShift Kubernetes Cloud services to deploy Kafka clusters using vendor provided helm charts & docker images
Kafka Resiliency dc/dr setup & observability for Kafka nodes
Data ingestion & streaming pipeline setup for other data platforms to Kafka & Kafka to data sources like Solr
Five or more years of Kafka administration experience, ideally with Confluent Kafka
Proficiency in Linux, Python, or Bash scripting
Hands-on knowledge of Kafka monitoring tools and experience with security protocols.
Thanks..
Kumar P
IT Analyst, Humac Inc
Phoenix, AZ 85027
Email id:
