Azure Data Engineer

Skills Needed: DevOps, Elasticsearch, Git, microservices, Linux, Microsoft AZURE, Big Data, Machine Learning, JSON, Restful APIs, PostgreSQL, Docker, data bricks, Azure data factory, Architecture, Apache Kafka, Microsoft SQL Server, backend ,Fast API, Distributed Systems, Data Pipelines ,Infrastructure Management, Artificial Intelligence, Low Latency, Relational Databases, Deployment Automation, Code Review, Large Language Models, Application Programming Interfaces (APIs),Cloud Engineering, Python (Programming Language),Production Systems, Software Engineering, Cloud Computing, Lifecycle Management, Databases, Health Care, Data Logging, Data Streaming, Extract Transform Load (ETL),Information Engineering, Cloud Computing Security, Data Lakes, GitHub Copilot, Software Deployment, Indexer, Enterprise Data Management, Data Ingestion, Enterprise Software Applications, Application Services, Event Driven Architecture, Mobile Application Development, Enterprise Search, Confluent, Atlassian Bamboo

Job Description:

Looking for a strong Azure Data Engineer / Senior Software Engineer with strong expertise in Python, Databricks, SQL, Elasticsearch, Kafka, and Azure, responsible for building scalable search platforms, REST APIs, event-driven data pipelines, and cloud-native distributed systems supporting high-volume enterprise applications.

Roles and Responsibilities:

• Resource will be responsible to design, build, and support client, our enterprise search and intelligent matching platform.
• This role will develop scalable APIs, high-performance search capabilities, event-driven data pipelines, and cloud-native services supporting millions of transactions daily.
• The consultant will work across Azure, Elasticsearch, Kafka, Databricks, Python, and distributed systems to build highly available, low-latency applications.
• The ideal candidate has strong backend engineering experience with cloud-native architectures, search technologies, REST APIs, event streaming, and large-scale data processing.
• Design and develop scalable backend services using Python. Build and maintain REST APIs supporting enterprise applications. Design and optimize Elasticsearch indexes, queries, mappings, and search relevance.
• Develop intelligent search and matching capabilities for large-scale datasets. Build event-driven architectures using Kafka and Azure Event Hub. Design high-throughput data ingestion and indexing pipelines. Integrate applications with Databricks and enterprise data platforms.
• Develop secure cloud-native applications on Microsoft Azure. Implement CI/CD pipelines and automated deployment processes.
• Monitor application health, logging, observability, and production performance. Optimize API response time, indexing performance, and infrastructure utilization. Participate in architecture, design reviews, code reviews, and production support.
• Collaborate with product managers, architects, data scientists, and engineering teams.

Educational Qualifications:

Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable.

Skills:

Mandatory skills:

• Programming Python (required) SQL REST APIs JSON FastAPI (preferred) Async Python Cloud Microsoft Azure Azure App Services Azure Linux VMs Azure Container Apps / AKS (preferred) Azure Managed Identity Azure Key Vault Azure Storage Azure Monitor Azure Event Hub Search
• Technologies Elasticsearch Query DSL Relevance tuning Mapping design Aggregations Multi-search (msearch) Bulk indexing Search optimization Index lifecycle management Event Streaming Apache Kafka Confluent Cloud Producers / Consumers Kafka Connect Schema Registry
• Event-driven architecture Data Engineering Databricks PySpark Delta Lake Data ingestion pipelines ETL/ELT DevOps Git Azure DevOps Azure Pipelines CI/CD Docker Infrastructure as Code (Terraform or Pulumi preferred) Databases PostgreSQL SQL Server Relational Databases Required
• Experience Min 8+ years of software engineering experience. Strong Python backend development experience. Experience designing and operating production REST APIs.
• Strong experience with Elasticsearch in production environments. Experience building scalable search platforms. Experience with Kafka or enterprise event-streaming platforms. Experience deploying applications on Microsoft Azure. Experience with CI/CD pipelines and automated deployments.
• Experience troubleshooting production systems and performance issues. Strong understanding of distributed systems and microservices.

Preferred Qualifications:

• Experience with AI-assisted search or intelligent matching platforms. Experience with Databricks and Delta Lake.
• Experience with machine learning integration. Experience with semantic search or vector search.
• Experience with Azure OpenAI or LLM integration. Healthcare or supply chain experience. Experience using GitHub Copilot or Codex.