Key skills
AWSPythonSQLDatabricksKubernetesVector databasesCI/CDContainerized applicationsInfrastructure-as-codeAIMilvusETLELTData EngineeringAnalyticsTerraformDockerEKSHelmCloudFormationLambdaS3RDSIAMSNSSQS
About this role
Harnham is an emerging AI-driven technology company seeking a Senior Data Engineer to own and operate the data infrastructure that powers their core products. This role involves building advanced systems for real-time decisioning, predictive analytics, and large-scale data processing for enterprise clients, blending backend engineering, cloud infrastructure, and data platform development.
Responsibilities:
- Design, build, and operate production data services and APIs in a cloud environment using containerized applications
- Implement and scale vector search capabilities supporting high‑volume similarity retrieval across 50M+ records
- Build and optimize data pipelines and ETL/ELT workflows using Python, SQL, and Databricks
- Architect cost‑effective cloud infrastructure supporting real‑time and batch workloads
- Collaborate cross‑functionally with data science and product teams to translate requirements into scalable solutions
- Own monitoring, observability, and service reliability for key data‑driven systems
- Improve internal tooling, infrastructure components, and engineering best practices
Requirements:
- 5+ years in data engineering, backend engineering, or platform infrastructure roles
- Strong hands‑on experience with AWS (EKS, S3, RDS, Lambda, IAM, SQS/SNS)
- Proficiency deploying and troubleshooting containerized applications on Kubernetes
- Production-grade Python and SQL experience
- Hands-on experience with Databricks (Delta Lake, Jobs, Workflows)
- Experience working with vector databases or vector search technologies (Milvus, Databricks Vector Index)
- Familiarity with CI/CD, Docker, Helm, and infrastructure-as-code (Terraform or CloudFormation)