AirflowAWSFluxPythonSparkSQLAIDatabricksIntegration TestingPerformance OptimizationRemote Work
About this role
Role Overview
Diagnose and resolve complex data processing issues to ensure the reliability and performance of critical data pipelines.
Collaborate with the team to support timely software delivery by proactively managing data-flow risks and dependencies.
Prevent data errors and inconsistencies by strengthening validation, monitoring, and anomaly detection across a complex data ecosystem.
Deliver additional business value each development cycle by shipping well-scoped improvements and fixes iteratively.
Ramp up quickly by learning the existing development cycle, following established processes, and collaborating effectively across time zones as needed.
Requirements
Hands-on expertise with Spark, Airflow, AWS, and the Databricks ecosystem is required.
Strong programming and query skills in Python and SQL; familiarity with Hive is a plus.
Deep understanding of big data processing concepts (trade-offs, limits, governance, and performance optimization).
Proven ability in debugging, unit and integration testing, managing task dependencies, monitoring, and anomaly detection.
Experience with Delta format and comfort using modern AI tools (e.g., Copilot, ChatGPT) to boost productivity.
Tech Stack
Airflow
AWS
Flux
Python
Spark
SQL
Benefits
Comprehensive health benefits, life and disability insurance, and fertility and family-building support programs.
Generous paid time off and holidays, volunteer leave, quarterly personal wellness days, and meeting-free days.
Tuition and book reimbursement programs to support continuous learning and professional development.
Thrive Global wellness program, confidential Employee Assistance Program (EAP), and one-on-one wellness coaching.
Employee programs — including Employee Resource Groups (ERGs), the “GoTo Gives Back” initiative, and our charitable donation matching program — to expand your social network and amplify the impact of your efforts.
A registered retirement savings plan to help you plan for the future.
Gym reimbursement programs to support your physical well-being.
Access to telemedicine services for convenient healthcare.
A GoTo performance bonus program to recognize your contributions.
A monthly remote work allowance to cover home office expenses.