Senior Technology Consultant - Data / AI Native Engineer

NTT DATA is seeking a hands-on Senior Data / AI Native Engineer with expertise in Databricks, AWS, PySpark, and Lakehouse architecture. The role leads and delivers enterprise data engineering initiatives, builds scalable data platforms and pipelines, and applies AI-assisted engineering practices to data workflows.

Responsibilities:

• Design, develop, and optimize high-volume enterprise data pipelines using PySpark, Spark, and Databricks
• Build scalable Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze/Silver/Gold)
• Implement data governance, access controls, and cataloging using Databricks Unity Catalog
• Design and develop cloud-native data solutions using AWS S3, EMR, Glue, Lambda, and Redshift
• Lead complex Data Engineering initiatives and provide technical guidance to engineering teams
• Develop batch and real-time data processing pipelines for large-scale enterprise datasets
• Use GitHub Copilot and/or Claude Code CLI daily to accelerate coding, pipeline development, testing, troubleshooting, and documentation
• Rapidly develop POCs and MVPs using AI-assisted engineering practices
• Apply AI within data pipelines for use cases such as schema inference, automated data-quality rule generation, anomaly detection, and PySpark transformation generation
• Build AI-ready data platforms supporting RAG, embeddings, vector stores, and AI/ML applications
• Develop streaming pipelines using Structured Streaming, Kafka, Kinesis, or Databricks Auto Loader
• Implement modern data engineering patterns including CDC, SCD Type 2, schema evolution, idempotent processing, and data contracts
• Implement data quality, validation, monitoring, lineage, and observability across data pipelines
• Conduct code/design reviews and establish reusable Data Engineering patterns and standards
• Collaborate with Data Architects, AI/ML Engineers, Software Engineers, and business stakeholders to deliver enterprise data products

Requirements:

• 7+ years of experience in Data Engineering and Software Engineering, building production-grade enterprise data solutions
• 4+ years of hands-on experience with Databricks, Delta Lake, Medallion Architecture, and PySpark/Spark
• 4+ years of experience with the AWS data ecosystem, including S3, EMR, Glue, Lambda, and/or Redshift
• Proven experience leading complex Data Engineering initiatives or providing technical leadership to Data Engineering teams
• Strong hands-on experience processing high-volume datasets using PySpark, rather than Scala-only Spark development
• Demonstrated daily use of GitHub Copilot and/or Claude Code CLI, with the ability to explain specific examples of how these tools improve Data Engineering productivity
• Proven experience rapidly developing POCs and MVPs using AI-assisted development practices
• Databricks, Delta Lake, Unity Catalog
• PySpark, Apache Spark
• AWS – S3, EMR, Glue, Lambda, Redshift
• Lakehouse, Medallion Architecture, Data Lakes
• Spark Structured Streaming, Kafka, Kinesis, Auto Loader