Key skills
AWSAzureCloudDockerKafkaKubernetesLinuxNGINXPostgresPySparkPythonTerraformShellBashSnowflakeGoogle CloudArgoCDPostgreSQLPerformance OptimizationLoad BalancingCI/CDProject Management
About this role
Role Overview
- Data Architecture Direction: Provide strategic direction for our data architecture, selecting the appropriate componments for various tasks. Collaborate on requirements and make final decisions on system design and implementation.
- Project Management: Manage end-to-end execution of high-performance, large-scale data-driven projects, including design, implementation, and ongoing maintenance.
- Cost Optimization: Monitor and optimize cloud costs associated with data infrastructure and processes.
- Efficiency and Reliability: Design and build monitoring tools to ensure the efficiency, reliability, and performance of data processes and systems.
- DevOps Integration: Implement and manage DevOps practices to streamline development and operations, focusing on infrastructure automation, continuous integration/continuous deployment (CI/CD) pipelines, containerization, orchestration, and infrastructure as code. Ensure scalable, reliable, and efficient deployment processes.
Requirements
- 5+ Years of Experience: Demonstrated experience as a DevOps professional, with a strong focus on big data environments, or Data Engineer with strong DevOps skills.
- Data Components Management: Experiences managing and designing data infrastructure, such as Snowflake, PostgreSQL, Kafka, Aerospike, and Object Store.
- DevOps Expertise: Proven experience creating, establishing, and managing big data tools, including automation tasks. Extensive knowledge of DevOps concepts and tools, including Docker, Kubernetes, Terraform, ArgoCD, Linux OS, Networking, Load Balancing, Nginx, etc.
- Programming Skills: Proficiency in programming languages such as Python and Object-Oriented Programming (OOP), emphasizing big data processing (like PySpark). Experience with scripting languages like Bash and Shell for automation tasks.
- Cloud Platforms: Hands-on experience with major cloud providers such as Azure, Google Cloud, or AWS.
- Performance Optimization: Experience in optimizing performance for big data tools and pipelines
- Big Advantage.
- Security Expertise: Experience in identifying and addressing security vulnerabilities within the data platform
- Big Advantage.
- CI/CD Pipelines: Experience designing, implementing, and maintaining Continuous Integration/Continuous Deployment (CI/CD) pipelines – Advantage.
- Data Pipelines: Experience in building big data pipelines
- Advantage.
Tech Stack
- AWS
- Azure
- Cloud
- Docker
- Kafka
- Kubernetes
- Linux
- NGINX
- Postgres
- PySpark
- Python
- Terraform