Key skills
Apache SparkScalaAzure Data LakeAzure DatabricksAzure Synapse AnalyticsAzure Data FactorySnowflakeCloud-based big data solutionsData modelingETL/ELT pipelinesDatabase conceptsSpark StreamingKafkaEvent HubsDelta LakeDevOps practicesCI/CD pipelinesPythonAIMachine LearningETLELTData EngineeringData LakeAnalyticsDatabricksSparkAzurePerformance OptimizationCI/CD
About this role
Elios AI is an award-winning company recognized for exceptional employee satisfaction and inclusivity. They are seeking a Senior Data Engineer to design and implement scalable data pipelines and optimize data workflows using cutting-edge technologies within the Azure ecosystem.
Responsibilities:
- Design and implement scalable, efficient data pipelines using Azure Data Lake, Databricks, Snowflake, and Synapse Analytics
- Develop and optimize workflows with Apache Spark and Scala for batch and streaming data processing
- Build, maintain, and enhance robust ETL/ELT pipelines tailored to big data applications within Azure ecosystems
- Manage and optimize data storage solutions like Azure Data Lake Storage, Snowflake, and Synapse Analytics to ensure peak performance and cost-efficiency
- Partner with data scientists, analysts, and business teams to ensure the reliability and availability of data platforms
- Monitor and fine-tune the performance of data platforms in production environments
- Enforce best practices for data governance, security, and compliance in Azure-based data frameworks
- Stay ahead of the curve by researching and integrating new big data technologies to enhance scalability and performance
Requirements:
- A minimum of 5 years of experience in data engineering or related fields
- At least 3 years of hands-on expertise in Apache Spark using Scala
- Advanced knowledge of Azure data services, including Azure Data Lake, Azure Databricks, Azure Synapse Analytics, and Azure Data Factory
- Proficiency with Snowflake, including schema design, performance optimization, and integration with cloud platforms
- Solid expertise in cloud-based big data solutions, particularly within the Azure ecosystem
- Strong knowledge of data modeling, ETL/ELT pipelines, and database concepts
- Experience with streaming platforms such as Spark Streaming, Kafka, or Event Hubs
- Familiarity with data lake and data warehouse architecture (e.g., Delta Lake, Snowflake, Synapse)
- Proficiency in DevOps practices, including CI/CD pipelines for data engineering workflows
- Knowledge of Python for data engineering tasks
- Experience with Azure Machine Learning or other machine learning platforms integrated with data workflows