Identify and acquire data sets that align with organisational business needs and strategic objectives
Establish and support the development of robust data streaming systems to enable real-time data processing and delivery
Develop comprehensive business intelligence reports and dashboards for executive leadership and company advisors
Develop sophisticated algorithms to transform raw data into actionable business insights
Establish, validate, and maintain optimal database pipeline architectures to support enterprise scale operations
Develop and establish innovative data validation methodologies and analysis tools to enhance data quality and insights
Maintain strict adherence to data governance frameworks and security policies across all data operations
Drive continuous improvement in data pipeline reliability, performance, and business value delivery

B.E/B.Tech/MS in Computer Science or related field with 5+ years of experience in large-scale distributed systems
Hands-on experience with big data offerings including GCP BigQuery and Spark Streaming
Hands-on experience with big data tools (Hadoop, Apache Spark, Hive, Hudi) and workflow management (Apache Airflow)
Familiarity with in-memory processing and modern data formats (Avro, Parquet, JSON) and open table formats like Hudi
Strong experience with SQL and NoSQL databases (Cassandra, Cosmos) and distributed SQL (Presto/Trino)
Experience with integration tools like Airflow and workflow automation
Expertise in querying and monitoring logs using Grafana and Splunk
Experience with CI/CD automation using Jenkins and GitHub Actions

Data Engineer III

Key skills