Staff Data Engineer

Panasonic Automotive North America is an industry-leading global supplier to Automotive Original Equipment Manufacturers. The Staff Data Engineer is responsible for leading the building and maintaining of data infrastructure, ensuring data availability, quality, and reliability while leading a team of offshore resources.

Responsibilities:

• Data Pipeline Architecture Design: responsible for designing and architecting data pipelines that move, transform, and process data from various sources to target data storage or analytics platforms and need to ensure scalability, reliability, and efficiency in these architectures
• Data Modeling: Leads the design and implementation of data models that suit the requirements of the project and structuring data to support efficient querying and analysis
• ETL (Extract, Transform, Load) Processes: Building robust ETL processes is a key responsibility. Staff Data Engineers lead the design and development of ETL workflows that extract data from source systems, transform it into the desired format, and load it into Azure Data Lake
• Data Integration: integrating data from different sources, which may include databases, APIs, third-party services, and more. Integration involves handling data in various formats and ensuring data consistency and accuracy
• Data Quality and Monitoring: implement data quality checks and monitoring mechanisms to identify and rectify data quality issues. Also create alerts and notifications for anomalies in data processing
• Performance Optimization: responsible for optimizing the performance of data pipelines and processing systems. This includes fine-tuning queries, optimizing data storage, and managing resources effectively to ensure timely data processing
• Data Security: Ensuring data security and implement security measures to protect sensitive data and ensure proper access controls
• Collaboration: collaborate with cross-functional teams, including data analysts, business analysts and software engineers, to understand data requirements and provide the necessary guidelines to support their work
• Technology Selection: assess and select appropriate tools, technologies, and frameworks for different aspects of the data engineering process and stay up to date with industry trends and emerging technologies
• Leadership and Mentoring: As senior members of the team, will take on leadership roles by guiding and mentoring other data engineers, and works with department management to define and communicate work duties and objectives to the individuals within the team. Lead or participate in code reviews, best practice discussions, and training sessions. Responsible for leading a team of offshore resources
• Troubleshooting and Support: When issues arise in data pipelines or systems, Senior Data Engineer is responsible for diagnosing and resolving these issues in a timely manner to minimize disruptions
• Documentation: document data pipelines, workflows, architecture designs, and other relevant processes to ensure knowledge sharing and maintainable systems

Requirements:

• A Bachelor's or Master's degree in computer science, information technology, data science, or a related field with Eight (8)+ years of experience as a data engineer, with a significant portion of that experience specifically working with Azure data services
• Expertise in Microsoft Azure services and tools related to data engineering, such as Azure Data Factory, Azure Databricks, Azure Analytical Service, Azure Synapse Analytics, Azure Cosmos DB, etc
• Strong Experience in implementing data solutions in Azure Data Lake platform
• Expertise in SQL, T-SQL and experience working with various database systems, including relational databases and NoSQL databases is required
• Experience with data modeling, schema design, and performance optimization, especially creating tabular models for Self-service analytics using Power BI
• Strong skills in data integration, ETL (Extract, Transform, Load) processes, and data orchestration workflows
• Expertise in data warehousing concepts and techniques
• Programming skills in languages such as Python, PySpark, Java, Scala, or PowerShell for scripting and automation
• Experience designing, building, and optimizing end-to-end data pipelines
• Proven track record of implementing data solutions that meet performance, scalability, and reliability requirements
• Experience with data migration, data warehousing, and data lake solutions
• Data accuracy and quality are paramount. A Senior Azure Data Engineer should be meticulous in ensuring that data pipelines and processes maintain high data integrity
• Experience with Data Governance tools like Azure Purview, Alation.etc are preferred
• Experience in creating data pipelines to extract the data from SAP applications is preferred