Lead Data Engineer

Quantiphi is an award-winning, AI-First global digital engineering company that helps leading organizations transform bold ideas into measurable business impact. The Lead Data Engineer will be responsible for building and maintaining scalable data pipelines and data warehouses, ensuring data quality, and translating business requirements into effective dashboards.

Responsibilities:

• Hands-on experience in building and maintaining scalable and cost-effective data pipelines, data warehouses and data lake using AWS S3, AWS Glue/Amazon EMR, AWS Lambda to ingest data from various sources (e.g., APIs, databases, flat files, etc.)
• Expertise in implementing data modeling and data warehousing concepts to create a well-structured analytical layer on top of the data lake, optimized for querying with AWS Redshift and AWS Athena
• Proficiency in SQL (complex joins, window functions, query optimization) and strong programming skills in Python/PySpark for data processing and automation
• Ensure data quality, integrity, and reliability by implementing robust monitoring, alerting, and validation frameworks for all data pipelines
• Solid understanding of data modeling concepts (e.g., star/snowflake schemas, SCD) and data warehousing principles
• Hands-on experience with Amazon QuickSight, with a strong understanding of the latest features and capabilities
• Ability to understand business requirements and translate them into scalable dashboards, including KPIs, tooltips, user interactions, dashboard navigation, personas, access management, and reporting requirements
• Strong experience in: SPICE optimization, Direct Query, Data blending, Custom visual creation, Calculated fields and parameters
• Experience implementing Row-Level Security (RLS) and other security controls within QuickSight
• Ability to independently design, develop, test, and deploy production-grade QuickSight dashboards
• Validate dashboard functionality, KPI calculations, and data accuracy against business requirements
• Experience documenting dashboard design, implementation, testing, validations, and deployment activities
• Ability to optimize queries, datasets, and visualizations for performance and scalability
• Experience developing and maintaining datasets and data models for Amazon QuickSight and other BI platforms
• Ability to gather reporting requirements from business stakeholders and convert them into meaningful dashboards and visualizations
• Strong verbal and written communication skills with the ability to communicate effectively with both business and technical teams

Requirements:

• Hands-on experience in building and maintaining scalable and cost-effective data pipelines, data warehouses and data lake using AWS S3, AWS Glue/Amazon EMR, AWS Lambda to ingest data from various sources (e.g., APIs, databases, flat files, etc.)
• Expertise in implementing data modeling and data warehousing concepts to create a well-structured analytical layer on top of the data lake, optimized for querying with AWS Redshift and AWS Athena
• Proficiency in SQL (complex joins, window functions, query optimization) and strong programming skills in Python/PySpark for data processing and automation
• Ensure data quality, integrity, and reliability by implementing robust monitoring, alerting, and validation frameworks for all data pipelines
• Solid understanding of data modeling concepts (e.g., star/snowflake schemas, SCD) and data warehousing principles
• Hands-on experience with Amazon QuickSight, with a strong understanding of the latest features and capabilities
• Ability to understand business requirements and translate them into scalable dashboards, including KPIs, tooltips, user interactions, dashboard navigation, personas, access management, and reporting requirements
• Strong experience in: SPICE optimization, Direct Query, Data blending, Custom visual creation, Calculated fields and parameters
• Experience implementing Row-Level Security (RLS) and other security controls within QuickSight
• Ability to independently design, develop, test, and deploy production-grade QuickSight dashboards
• Validate dashboard functionality, KPI calculations, and data accuracy against business requirements
• Experience documenting dashboard design, implementation, testing, validations, and deployment activities
• Ability to optimize queries, datasets, and visualizations for performance and scalability
• Experience developing and maintaining datasets and data models for Amazon QuickSight and other BI platforms
• Ability to gather reporting requirements from business stakeholders and convert them into meaningful dashboards and visualizations
• Strong verbal and written communication skills with the ability to communicate effectively with both business and technical teams
• Experience with workflow orchestration tools like AWS Step Functions or Apache Airflow
• Experience managing QuickSight assets, including publishing dashboards, scheduling refreshes, user administration, and access management
• Experience with Tableau or Power BI
• Proven experience migrating dashboards and reports from BI platforms such as Qlik, Tableau, or Power BI to Amazon QuickSight
• Experience monitoring and supporting analytics solutions in AWS environments