Senior Data Engineer

Panasonic Automotive North America is a leading global supplier in the automotive industry, specializing in infotainment systems and advanced connected car solutions. The Senior Data Engineer will build and maintain the data infrastructure to support efficient data collection, processing, and analysis, ensuring data availability, quality, and reliability throughout its lifecycle.

Responsibilities:

• Data Pipeline Architecture Design: executes the architecting and supports the design of data pipelines that move, transform, and process data from various sources to target data storage or analytics platforms and need to ensure scalability, reliability, and efficiency in these architectures
• Data Modeling: Design and implement data models that suit the requirements of the project and structuring data to support efficient querying and analysis
• ETL (Extract, Transform, Load) Processes: Building robust ETL processes is a key responsibility. Senior Data Engineers design and develop ETL workflows that extract data from source systems, transform it into the desired format, and load it into Azure Data Lake
• Data Integration: integrating data from different sources, which may include databases, APIs, third-party services, and more. Integration involves handling data in various formats and ensuring data consistency and accuracy
• Data Quality and Monitoring: implement data quality checks and monitoring mechanisms to identify and rectify data quality issues. Also create alerts and notifications for anomalies in data processing
• Performance Optimization: responsible for optimizing the performance of data pipelines and processing systems. This includes fine-tuning queries, optimizing data storage, and managing resources effectively to ensure timely data processing
• Data Security: Ensuring data security and implement security measures to protect sensitive data and ensure proper access controls
• Collaboration: collaborate with cross-functional teams, including data analysts, business analysts and software engineers, to understand data requirements and provide the necessary guidelines to support their work
• Technology Selection: assess and select appropriate tools, technologies, and frameworks for different aspects of the data engineering process and stay up to date with industry trends and emerging technologies
• Leadership and Mentoring: As senior members of the team, should be able take on project leadership roles by guiding and mentoring other data engineers. Participate in code reviews, best practice discussions, and training sessions
• Troubleshooting and Support: When issues arise in data pipelines or systems, Senior Data Engineer is responsible for diagnosing and resolving these issues in a timely manner to minimize disruptions

Requirements:

• A Bachelor's or Master's degree in computer science, information technology, data science, or a related field with Five (5)+ years of experience as a data engineer, with a significant portion of that experience specifically working with Azure data services
• Proficiency in Microsoft Azure services and tools related to data engineering, such as Azure Data Factory, Azure Databricks, Azure Analytical Service, Azure Synapse Analytics, Azure Cosmos DB, etc
• Strong Experience in implementing data solutions in Azure Data Lake platform
• Expertise in SQL, T-SQL and experience working with various database systems, including relational databases and NoSQL databases is required
• Experience with data modeling, schema design, and performance optimization, especially creating tabular models for Self-service analytics using Power BI
• Strong skills in data integration, ETL (Extract, Transform, Load) processes, and data orchestration workflows
• Expertise in data warehousing concepts and techniques
• Programming skills in languages such as Python, PySpark, Java, Scala, or PowerShell for scripting and automation
• Experience designing, building, and optimizing end-to-end data pipelines
• Proven track record of implementing data solutions that meet performance, scalability, and reliability requirements
• Experience with data migration, data warehousing, and data lake solutions
• The ability to analyze complex data engineering challenges and develop innovative solutions
• Strong troubleshooting skills to identify and resolve issues in data pipelines and systems
• Effective communication skills to work collaboratively with cross-functional teams, including data scientists, analysts, and business stakeholders
• The ability to translate technical concepts into understandable terms for non-technical team members
• Data accuracy and quality are paramount. A Senior Azure Data Engineer should be meticulous in ensuring that data pipelines and processes maintain high data integrity
• As a senior role, this individual might be required to lead and mentor junior members of the data engineering team
• Experience with Data Governance tools like Azure Purview, Alation.etc are preferred
• Experience in creating data pipelines to extract the data from SAP applications is preferred