Key skills
Data pipeline designDockerKubernetesApache AirflowPythonApache SparkGoogle Cloud servicesTerraformDaskGoRustRayData governanceRole-based access controlAIMLData EngineeringSparkAirflowGCPGoogle CloudGKEHelmPub/SubCloud StorageCI/CD
About this role
Versa Networks is revolutionizing the way businesses connect and secure their networks, and they are seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines. The role involves collaborating with data scientists and software engineers to ensure scalable and reliable data infrastructure.
Responsibilities:
- Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases
- Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts
- Build high-throughput data transformations using Dask or Apache Spark
- Maintain training data clusters across hybrid (on-prem and cloud environments)
- Optimize training jobs for performance, resiliency, and cost
- Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence
- Troubleshoot, debug, and resolve data-processing failures in production
- Work with cross-functional teams to define data contracts, schemas, and quality checks
- Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation
- Design and maintain data models and schemas for AI/ML continuous training use cases
- Load data into cloud storage and lakes, ensuring performance and accessibility
Requirements:
- 3–5 years of professional experience designing and operating production data pipelines at scale
- Containerization & Orchestration: Expertise with Docker, Kubernetes, and Helm
- Workflow Management: Hands-on experience building DAG-based pipelines in Apache Airflow
- Programming: Strong proficiency in Python for data engineering tasks
- Distributed Frameworks: Practical experience with Dask or Apache Spark for large-scale data processing
- Cloud Fundamentals: Familiarity with deploying and managing services in a cloud environment
- Compiled Languages: Experience writing data services in Go or Rust
- GCP Proficiency: Hands-on with Google Cloud services (e.g., Pub/Sub, Big Query, Cloud Storage, GKE). Equivalent experience in other public cloud providers is fine
- ML Pipelines: Exposure to deploying cross-cluster model-training workflows using Ray or similar frameworks
- Infrastructure as Code: Familiarity with Terraform for deployment
- Security & Compliance: Knowledge of data governance, encryption, and role-based access control
- Applicants must be authorized to work in the US