Data Engineer III- Remote

ConsultNet Technology Services and Solutions is recruiting a Data Engineer III for a remote contract role. The position independently designs, develops, and maintains complex data ingestion pipelines into Snowflake, integrating sources such as Kafka, relational databases, and mainframe files while applying data obfuscation, quality controls, and established SDLC and CI/CD processes.

Responsibilities:

• Partner with the Digital team to design, develop, and maintain data ingestion pipelines into Snowflake
• Build pipelines supporting Kafka, relational databases, mainframe flat files, and other data sources
• Develop ETL processes using Qlik or comparable data integration technologies
• Implement data obfuscation within the ETL process to protect sensitive information
• Move processed data through S3 into Snowflake
• Load data into appropriate DDL/schema structures using Spark within Snowflake
• Develop controls to validate data ingestion and quality
• Monitor streaming data, including Kafka topics, to ensure data is loading and processing correctly
• Troubleshoot pipeline, data-quality, and ingestion issues
• Perform QA/QC throughout the development lifecycle
• Follow established software development, CI/CD, and SDLC processes
• Document development and technical changes

Requirements:

• 8 years of professional experience
• Senior-level Data Engineering experience
• Strong software development and SDLC knowledge
• Ability to work independently with minimal supervision
• Git and experience working collaboratively within a shared repository
• CI/CD experience
• Strong data ingestion and pipeline development experience
• ETL tool experience — required
• Specific ETL platform is not required
• Experience with tools such as Databricks or similar platforms is transferable
• Qlik Replicate is not required and can be learned on the job
• Experience working with multiple data sources, including:
• Kafka
• Relational databases/views
• Mainframe flat files
• Change Data Capture (CDC)
• AWS Lambda
• AWS S3
• Snowflake
• Snowflake stored procedures (Sprocs)
• Spark within Snowflake
• User-defined functions (UDFs)
• Data obfuscation and protection of highly sensitive data
• Experience in banking, financial services, or another regulated environment is helpful but not required
• Experience working with complex, sensitive, or highly regulated data is beneficial