Develop and maintain scalable data pipelines using tools like Apache Kafka, Apache Spark, or AWS Glue
Design and implement data models and schemas in data warehouses (e.g., Amazon Redshift, Google BigQuery)
Collaborate with data scientists and analysts to understand data requirements
Utilize ETL tools and frameworks (e.g., Apache NiFi, Talend, or custom Python scripts)
Monitor and optimize data pipeline performance using tools like Apache Airflow or AWS Step Functions
Write, test, and maintain scripts in Python, SQL, or Bash
Implement CI/CD practices for data engineering workflows using tools like Jenkins, GitLab CI, or Azure DevOps
Collaborate with DevOps teams to integrate data solutions into existing infrastructure
Manage containerized data applications using Docker and orchestrate them with Kubernetes
Implement monitoring and logging solutions using tools like Prometheus, Grafana, or ELK Stack
Ensure compliance with data governance, security best practices, and data privacy regulations

Expert proficiency in Terraform
Extensive experience across at least two major cloud platforms (AWS, Azure, GCP)
Strong hands-on experience with Kubernetes, Helm charts, and designing/optimizing CI/CD pipelines
Proficiency in Python and scripting languages (Bash/PowerShell)
Experience with data-related tools and technologies, including Apache Kafka, Apache Spark, or AWS Glue
Familiarity with data storage solutions like Amazon Redshift, Google BigQuery, or Azure Data Lake
Experience leading cloud migration initiatives
Excellent problem-solving, communication, and collaboration skills
Experience with configuration management tools (Ansible, Puppet) and a solid understanding of DevSecOps principles
Experience with OpenShift is a plus

Senior Data Engineer

Key skills