Implement data validation, cleansing, and monitoring techniques to ensure that the data is accurate, reliable, and up to date.
Define data models and schemas that support the integration and scalability of various data systems.
Monitor the health of data pipelines, troubleshoot issues, and improve system resilience to maintain data flow continuity.
Stay up to date with emerging technologies, frameworks, and industry trends, driving the adoption of new tools and methodologies.
Collaborate closely with Data Engineering and other key stakeholders to develop and execute a strategy to modernize data pipelines to address emerging needs for the business.
Implement and maintain robust security protocols, ensuring data privacy and compliance with relevant regulations.
Follow and promote best practices in coding, design, and development methodologies, maintaining quality and consistency across the team.
Provide guidance support and mentorship, fostering a collaborative and growth-oriented environment.
Conduct code reviews and offer constructive feedback to ensure code quality, maintainability, and system scalability.
Requirements
Bachelor’s degree in Software Engineering or related field.
5+ years of experience as a Data Engineer or in a similar role, with a strong focus on building data pipelines and working alongside application engineering teams, with programming expertise in Python, .NET, and Spark technologies
Experience with cloud data platforms and data warehouses, specifically experience with Snowflake is required
Hand on experience with building real-time continuous data streaming platforms, utilizing various modern data tooling technologies (e.g., Snowflake, Kafka, Kinesis)
Proficiency in SQL and experience with modern data ETL/ELT and orchestration tools, i.e., DBT, Airflow, Fivetran, etc.
Extensive SnowSQL experience in developing stored procedures, writing queries to analyze and transform data.
Building out security and governance frameworks in Snowflake or other cloud native solutions.
Experience in Healthcare and claims processing is highly desirable.
Understanding of RESTful APIs and experience in developing or consuming APIs for data integration
CI/CD Pipeline and code management experience.
Detail-oriented, with a focus on code quality, maintainability, and adherence to best practices.
Excellent communication skills, with the ability to advise and guide others effectively.
Tech Stack
Airflow
Cloud
ETL
Kafka
Python
Spark
SQL
.NET
Senior Data Engineer at Lyric - Clarity in motion. | JobVerse