Data Engineer

Reality Defender is an award-winning cybersecurity company helping enterprises and governments detect deepfakes and AI-generated media. The Data Engineer will build and scale the infrastructure for the data platform, designing and operating pipelines for large datasets while collaborating with ML engineers and researchers.

Responsibilities:

• Design, build, and operate large-scale data processing pipelines handling multi-terabyte and streaming datasets, including audio/video transcoding, feature extraction, and preprocessing workflows
• Deploy, scale, and troubleshoot containerized workloads on Kubernetes and AWS in production environments
• Build and maintain distributed data processing jobs using frameworks such as Spark and Ray
• Design and operate workflow orchestration systems (e.g., Airflow) with dependency management, retries, monitoring, and alerting for production pipelines
• Administer and tune enterprise databases, including performance tuning, backup/recovery, access control, and scaling strategies
• Partner with ML engineers and researchers to support training pipelines, model retraining triggers, feature stores, and other MLOps workflows

Requirements:

• Hands-on experience with Kubernetes and AWS, including deploying, scaling, and troubleshooting containerized workloads in production environments
• Proficiency with high-performance/distributed computing frameworks such as Spark and Ray for processing large-scale datasets
• Experience with workflow orchestration tools such as Airflow (or comparable systems like Dagster, Prefect, or Luigi) to schedule and manage complex data pipelines
• Strong programming skills in Python and SQL
• Demonstrated track record building and operating large-scale data processing pipelines, ideally handling multi-terabyte or streaming datasets
• Familiarity with common data transformation patterns applied to large datasets (ETL/ELT, batch and stream processing, data validation and quality checks)
• Experience designing and maintaining job orchestration systems, including dependency management, retries, monitoring, and alerting for production pipelines
• Experience with Golang is a plus
• Experience working with audio or video data at scale is a strong plus (e.g., transcoding, feature extraction, or preprocessing pipelines)
• Bonus: experience orchestrating machine learning workflows (training pipelines, model retraining triggers, feature stores, or MLOps tooling)