Principal Software Engineer, Data Processing OSS

DataPelago is building a universal data processing engine that accelerates structured and unstructured data processing across parallel processing frameworks and infrastructure. The Senior/Principal Data Processing Engineer - OSS will advance the engine’s integration with open-source platforms such as Apache Gluten, Velox, Apache Spark, and Apache Flink, while improving functionality, performance, scalability, and reliability through design, development, optimization, and community collaboration.

Responsibilities:

• Architect: Influence the architecture of how our data processing engine interfaces with open-source platforms and engines
• Design: Lead design of functional and performance enhancements to open-source platforms such as Apache Gluten and Velox and their integration with our data processing engine
• Core Development: Individually design, implement, test, optimize, and maintain components of the data processing engine
• Innovation and Differentiation: Analyze technology roadmap of Apache Gluten, Velox, and equivalent platforms and identify opportunities for our engine to enhance technology and product leadership
• Collaboration: Partner effectively with engineering, product management, open-source community and customer success teams
• Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity

Requirements:

• 15+ years of relevant experience
• Bachelor's degree in Computer Science, or a related field OR a Master's degree in Computer Science or a related field
• 3+ years of deep technical experience in developing core components of Apache Spark, Apache Flink, Apache Doris, Apache Gluten, Velox, Apache DataFusion, Apache DataFusion Comet, or equivalent platforms designed for large-scale data processing
• 3+ years of deep technical experience in instrumenting, analyzing, and optimizing the performance of data processing engine components on benchmark and customer workloads
• Sound knowledge of the architecture and internal operation of one or more of Apache Spark, Apache Flink, Presto/Trino
• Demonstrated experience in the design, development, and successful release of high-performance data processing engines for large production deployments
• Exceptional programming skills in C, C++, and Java
• Extensive development experience in Linux environments
• Strong analytical and problem-solving skills with a passion for performance optimization