Key skills
SparkPySparkSQLPythonHiveData ModellingETL PipelinesPerformance TuningDistributed ComputingDebuggingCloud PlatformsAirflowScalaData GovernanceMachine LearningContinuous IntegrationRelevant CertificationsETLData EngineeringData WarehousingAnalyticsAWSAzure
About this role
Brillio is a technology consulting company, and they are seeking a Senior Software Development Engineer specializing in data engineering. The role involves designing and developing scalable data processing solutions, implementing robust data models, and collaborating with stakeholders to translate requirements into technical solutions.
Responsibilities:
- Design and develop scalable data processing solutions using Spark and PySpark to support advanced analytics initiatives
- Implement robust data models and optimize SQL queries for efficient data retrieval and transformation
- Collaborate with data analysts and business stakeholders to translate requirements into technical solutions
- Build and maintain ETL pipelines leveraging Python, Hive, and SQL for large-scale data integration
- Conduct thorough code reviews, performance tuning, and debugging to ensure high-quality deliverables
- Monitor, troubleshoot, and resolve issues in production data workflows, ensuring data accuracy and reliability
- Document technical processes, data models, and workflow architectures to facilitate knowledge sharing
- Stay updated with industry trends in big data technologies and proactively recommend improvements
Requirements:
- Experience Range: 2 - 4 years of experience in software development, with hands-on expertise in data analysis and engineering
- Advanced proficiency in Spark and PySpark
- Strong knowledge of SQL (basic and advanced)
- Expertise in Python programming for data processing
- Experience with Hive for data warehousing solutions
- Solid understanding of data modelling fundamentals
- Ability to design and optimize ETL pipelines
- Hands-on experience with large-scale data processing
- Proficient in performance tuning of SQL and Spark jobs
- Familiarity with distributed computing concepts
- Competence in debugging and troubleshooting data workflows
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
- Experience with cloud-based data platforms such as AWS or Azure
- Knowledge of Airflow or similar workflow orchestration tools
- Familiarity with Scala for Spark development
- Understanding of data governance and security best practices
- Exposure to machine learning frameworks in Python
- Experience with continuous integration and deployment for data applications
- Relevant certifications in big data technologies or data engineering (such as Spark, Python, or SQL) are advantageous