Senior Cloud Data Engineer

National Computer Systems is seeking a Senior Cloud Data Engineer for a cloud data modernization project. The role focuses on migrating on-premises ETL processes to Azure Cloud and Snowflake, building scalable data platforms and pipelines, and implementing AI/ML-enabled data engineering and MLOps solutions.

Responsibilities:

• Lead the migration of the ETLs from on-premises SQLServer based data warehouse to Azure Cloud and Snowflake
• Design, develop, and implement data platform solutions using Databricks, Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, Azure Data Lake Storage Gen2 (ADLS Gen2), Blob Storage, and Snowflake
• Review and analyze existing on-premises ETL processes developed in SSIS and T-SQL
• Implement DevOps practices and CI/CD pipelines using GitActions
• Collaborate with cross-functional teams to ensure seamless integration and data flow
• Optimize and troubleshoot data pipelines and workflows
• Ensure data security and compliance with industry standards
• Design and implement AI/ML-enabled data pipelines to improve data quality, anomaly detection, classification, forecasting, and operational insights
• Leverage Databricks Machine Learning, MLflow, and cloud-native AI services to support machine learning workflows
• Integrate Generative AI capabilities, Large Language Models (LLMs), and Azure OpenAI services into data engineering processes
• Develop automated solutions for metadata management, data cataloging, code generation, data validation, and documentation using AI technologies
• Build scalable feature engineering pipelines to support model training and inference workloads
• Collaborate with Data Scientists and AI Engineers to operationalize machine learning models within enterprise data platforms
• Implement MLOps practices for model versioning, deployment, monitoring, governance, and lifecycle management

Requirements:

• Minimum of 6+ years of experience as a Cloud Data Engineer
• Hands-on experience with Databricks, Azure Cloud data tools (ADF, SHIR, Logic Apps, ADLS Gen2, Blob Storage) and Snowflake
• Strong experience in ETL development using on-premises databases and ETL technologies
• Experience with Python or other scripting languages for data processing
• Proficiency in DevOps and CI/CD practices using GitActions
• Experience with Agile methodologies
• Excellent problem-solving skills and ability to work independently
• Strong communication and collaboration skills
• Strong analytical skills and attention to detail
• Ability to adapt to new technologies and learn quickly
• Experience with the application of AI/ML tools and models to data processing and ETL workloads
• Design and implement AI/ML-enabled data pipelines to improve data quality, anomaly detection, classification, forecasting, and operational insights
• Leverage Databricks Machine Learning, MLflow, and cloud-native AI services to support machine learning workflows
• Integrate Generative AI capabilities, Large Language Models (LLMs), and Azure OpenAI services into data engineering processes
• Develop automated solutions for metadata management, data cataloging, code generation, data validation, and documentation using AI technologies
• Build scalable feature engineering pipelines to support model training and inference workloads
• Collaborate with Data Scientists and AI Engineers to operationalize machine learning models within enterprise data platforms
• Implement MLOps practices for model versioning, deployment, monitoring, governance, and lifecycle management
• Experience with data modeling and database design
• Knowledge of data governance and data quality best practices
• Experience with development in Databricks for data engineering and analytics workloads
• Familiarity with other cloud platforms (e.g., AWS, Google Cloud)
• Certification in Azure or Snowflake