Data Engineer

BAE Systems is a defense, aerospace, and security company. The Senior Data Engineer designs and maintains data warehouses, ETL pipelines, and AI/ML models, collaborates with customers and cross-functional teams, and provides technical support for Amazon Redshift.

Responsibilities:

• Design and implement data pipelines to ingest, extract, transform, load (ETL) data and store large data sets from various sources
• Build and maintain data warehouses, including data modeling, data governance, and data quality
• Ensure data quality, integrity, and security by implementing data validation, data cleansing, and data governance policies
• Optimize data systems for performance, scalability, and reliability
• Collaborate with customers to understand their technical requirements and provide guidance on best practices for using Amazon Redshift
• Work with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver data solutions
• Provide technical support for Amazon Redshift, including troubleshooting, performance optimization, and data modeling
• Integrate AI/ML models with ETL pipelines and data warehouses to enable predictive analytics and machine learning
• Develop and deploy AI/ML models using popular frameworks such as TensorFlow, PyTorch, or Scikit-learn
• Work with data scientists to develop and deploy data-driven products and features
• Identify and resolve data-related issues, including data pipeline failures, data quality issues, and performance bottlenecks
• Develop technical documentation and knowledge base articles to help customers and AWS engineers troubleshoot common issues

Requirements:

• Bachelor's or Master's degree in Computer Science or a related field, with at least 6 years of experience in Information Technology, or equivalent combination of education and work experience
• Proficiency in one or more programming languages (e.g., Python, Java, Scala)
• 6+ years of experience in data engineering, with a focus on designing and implementing large-scale data systems
• 5 + years of hands-on experience in writing complex, highly optimized queries across large data sets using AWS Redshift, Oracle and SQL Server
• 5 + years of hands-on experience using AWS Glue, python/spark to build ETL pipelines in a production setting, including writing test cases
• Strong understanding of database design principles, data modeling, and data governance
• Proficiency in SQL, including query optimization, indexing, and performance tuning
• Experience with data warehousing concepts, including star and snowflake schemas
• Strong analytical and problem-solving skills, with the ability to break down complex problems into manageable components
• Experience with data storage solutions such as relational databases (Oracle, SQL Server), NoSQL databases and cloud-based data warehouses (Redshift)
• Experience with data processing frameworks such as Apache Kafka, Fivetran
• Experience in building ETL pipelines using AWS Glue, Apache Airflow, and programming languages including Python and PySpark
• Understanding of data quality and governance principles and best practices
• Experience with AWS services and best practices
• Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams
• Strong problem-solving and analytical skills, with the ability to break down complex problems into manageable components
• Ability to work in a fast-paced environment and adapt to changing priorities
• Strong attention to detail and ability to maintain high-quality documentation
• Commitment to continuous learning and professional development
• Experience with Dataiku
• Experience with building reports using PowerBI and Tableau
• Familiarity with agile development methodologies and version control systems such as Git
• Certification in data warehousing, ETL, or AI/ML (e.g., Certified Data Warehouse Architect, Certified ETL Developer)
• Experience with agile development methodologies such as Scrum or Kanban