ValueMomentumRemote
Senior Data Engineer
United States of America
Full Time
2 weeks ago
H1B Sponsor Likely
Key skills
Apache SparkDatabricksPySparkSpark SQLPythonAWSData EngineeringDistributed Data PipelinesData ModelingDistributed SystemsData ArchitectureMachine Learning PipelinesCSQLAIMachine LearningMLAnalyticsSparkCaching
About this role
ValueMomentum is a product development company with over 25 years in the market, specializing in P&C insurance. They are seeking a Senior Data Engineer to design, optimize, and operate large-scale Spark data platforms that support analytics and machine learning use cases.
Responsibilities:
- Design, build, and maintain distributed data pipelines using Apache Spark on Databricks
- Lead development of high‑performance Spark jobs for batch and near‑real‑time processing
- Implement complex data transformations using PySpark (DataFrame & Spark SQL APIs)
- Apply Spark best practices for partitioning, caching, joins, and shuffles
- Tune Spark jobs for performance, scalability, and cost efficiency
Requirements:
- 12+ years of experience in Data Engineering
- Strong hands‑on experience building and operating Apache Spark applications
- Proven experience running Spark workloads in Databricks (production environments)
- Advanced proficiency in PySpark, Spark SQL, and Python
- Experience working with large‑scale, distributed datasets in cloud environments (AWS preferred)
- Demonstrated experience supporting ML or AI pipelines on Spark
- Strong understanding of data modeling, distributed systems, and data architecture
- Experience designing reliable, automated, and observable Spark pipelines