Data Engineer

LanceSoft, Inc. is seeking a Data Engineer to design and implement data pipelines in support of RAG-based retrieval and reasoning. The role requires collaboration within a fast-moving team and proficiency in the Microsoft Azure environment.

Responsibilities:

• Build batch/streaming ingestion pipelines (ADF/Synapse/Databricks Workflows, Event Hubs/Kafka)
• Implement change data capture (CDC) patterns and schema evolution handling with quarantine controls
• Support automation of data reliability assessment
• Establish continuous integration/continuous deployment (CI/CD) pipelines, along with infrastructure-as-a-code orchestration (Terraform or Bicep), and segregated environment strategy (dev/test/prod)
• Define service level agreements (SLAs) and service level objectives (SLOs) for data reliability; configure monitoring & alerting
• Deliver cost optimization, performance tuning (OPTIMIZE, Z-ORDER), Backup strategy, and Disaster Recovery strategy
• Document architecture, patterns, and runbooks; conduct training for data engineers
• Collaborate with security, networking, and BI teams; drive stakeholder alignment

Requirements:

• 5–8 years in data engineering and data pipeline development
• Proficient in Python and SQL; experience with CI/CD and IaC (Terraform or Bicep)
• Excellent communication, documentation, and stakeholder management
• Expertise in engagement with business users to align business objectives with technical considerations within Azure