Data Engineer (remote US)

HarperCollins Publishers is seeking a Data Engineer to develop, maintain, and support enterprise data integration processes and pipelines across on-premises SQL Server and Microsoft Azure environments. The role supports modernization of legacy data processing into cloud-based architecture and collaborates with Data Governance, business stakeholders, application teams, and engineering resources to deliver reliable, well-documented data solutions.

Responsibilities:

• Design, develop, and maintain data pipelines using Microsoft Fabric, Azure Data Factory, and related Azure technologies
• Develop Python, PySpark, and SQL notebooks for data ingestion, transformation, and processing across Bronze, Silver, and Gold layers
• Integrate data from multiple sources, including files, databases, APIs, and enterprise applications
• Implement data transformations, enrichment, aggregations, and business rules based on approved specifications
• Optimize pipelines and SQL queries for performance, reliability, and scalability
• Develop and maintain enterprise Data Products in collaboration with Data Governance
• Maintain on-premises SQL Server databases, stored procedures, and ETL processes while supporting their migration to Azure and Microsoft Fabric
• Develop and maintain automated, rule-based processing workflows, including queue-driven processing and batch orchestration
• Manage file-based ingestion, feed tracking, processing status, and downstream data distribution
• Implement data validation, reconciliation, logging, monitoring, and exception handling
• Monitor production pipelines, troubleshoot failures and data discrepancies, and implement corrective actions
• Ensure continuity, reliability, and timely delivery of business-critical data processes
• Collaborate with business stakeholders and Data Governance to understand requirements, data definitions, business rules, and approved Data Product specifications
• Translate business requirements into technical designs, identifying source data gaps, dependencies, and process improvements
• Maintain technical documentation, including data flows, source-to-target mappings, transformation logic, and operational procedures
• Perform unit and integration testing, support QA and business sign-off, and adhere to established development, deployment, and Data Governance standards

Requirements:

• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related technical discipline
• 3–5+ years of hands-on experience in data engineering, database development, or enterprise data integration in building pipelines, migrating data and modernization of environments
• Strong proficiency in SQL Server, T-SQL, stored procedures, query optimization, and ETL/ELT development
• Hands-on experience with Azure Data Factory, Azure Data Lake Storage, and Microsoft Fabric, including Data Pipelines, Lakehouse, and Notebooks
• Proficiency in Python and/or PySpark, with an understanding of data warehousing, relational data modeling, and Medallion Architecture
• Experience with file-based integrations, batch processing, queue-driven workflows, data validation, monitoring, troubleshooting, and production support
• Strong communication and collaboration skills, with the ability to engage business stakeholders, Data Governance, and technical teams to translate requirements into effective solutions
• Strong analytical and problem-solving abilities, with attention to data accuracy, quality, and operational reliability
• Familiarity with SSIS or legacy ETL tools is an asset
• Experience modernizing legacy data platforms and relevant Microsoft certifications are considered assets
• cover letters are strongly preferred