Key skills
PythonApache SparkData pipelinesPerformance optimizationTime series dataDebugging skillsML workflowsStreaming systemsCloud-native platformsScalaMLSparkKafkaKubernetesPerformance Optimization
About this role
BayOne Solutions is seeking a Data Engineer to build and scale distributed data pipelines for large-scale time series and log data. The role involves designing high-performance workflows, improving data quality for ML workloads, and optimizing application performance in production systems.
Responsibilities:
- Build and scale distributed data pipelines for large-scale time series and log data
- Design reliable, high-performance Spark/Python workflows for model training datasets
- Analyze and resolve performance bottlenecks (latency, memory, skew, throughput)
- Improve data quality, validation, and reproducibility for ML workloads
- Partner with ML engineers and researchers to accelerate foundation model development
- Measure and optimize application and transaction performance in production systems
Requirements:
- 5+ years of software engineering experience
- Strong proficiency in Python
- Hands-on experience with Apache Spark (PySpark or Scala)
- Experience building large-scale data pipelines in distributed environments
- Experience working with time series, logs, or high-volume event data
- Strong debugging and performance optimization skills
- Experience supporting ML or large model training workflows
- Familiarity with sequence modeling or time series data systems
- Experience with streaming systems (Kafka, Spark Streaming)
- Experience with cloud-native or Kubernetes-based platforms