Key skills
ScalaSQLMachine LearningETLData WarehousingAnalyticsSparkApache SparkAirflowdbtAWSAzureGoogle CloudRDSPostgresLoad BalancingAgileScrumCommunication
About this role
CORMAC is seeking a skilled Data Engineer to manage and optimize data for powerful analytics and reporting. The role involves building and maintaining scalable data processing pipelines while ensuring data integrity, security, and accessibility for business-critical applications.
Responsibilities:
- Develop and maintain data processing applications using Apache Spark and Scala to process large-scale datasets efficiently
- Monitor and optimize database performance, including indexing, query optimization, and load balancing, to ensure high availability and low-latency data access
- Design and implement robust, scalable, and efficient database structures to support applications and data analytics
- Deploying and managing data infrastructure on cloud platforms like AWS, Azure, or Google Cloud
- Assist in the migration of data between systems, including database upgrades, platform changes, and ETL processes for data integration
- Leveraging cloud-native services for data processing and storage
- Optimizing costs while maintaining performance and scalability
- Applying various transformations like data cleansing, aggregation, and enrichment
- Monitor data quality metrics and perform data cleansing as necessary
- Work with Product Owners and Business Analysts to refine user stories for data ingestion, processing, and reporting features
Requirements:
- Bachelor's in Computer Science, Computer Engineering, Information Systems, or related field
- 3+ years of experience as a Data Engineer or similar role
- 2+ years of experience working with Apache Spark, Scala, and Spark SQL
- Must have the ability to obtain a Public Trust (Tier 1) Security Clearance
- Experience with cloud platforms such as AWS, Google Cloud, or Azure, utilizing cloud services for data storage, processing, and retrieval
- Knowledge of data warehousing, data lakes, and cloud computing
- Familiarity with machine learning algorithms
- Knowledge of ETL processes and data modeling techniques
- Familiarity with ETL tools, REST APIs, and data integration techniques
- Understanding of data security and compliance best practices
- Ability to develop clear, efficient, scalable solutions for complex problems
- Understanding of data warehousing solutions and database architecture principles
- Effective communication and teamwork abilities
- Strong problem-solving skills and attention to detail
- Experience using Airflow, DBT, Postgres, Amazon EMR & RDS
- Working experience on an Agile/Scrum or SaFE development team