Senior Software Engineering Advisor

Role Overview

• Partner with business and technical stakeholders to deeply understand and translate requirements into scalable data solutions.
• Design and document robust, high-performance data architectures that support strategic business outcomes.
• Build and optimize production-grade ETL pipelines using Spark and PySpark, ensuring reliability and efficiency.
• Develop and maintain data models that enable advanced analytics and reporting.
• Write efficient, complex SQL queries (Teradata SQL, Hive SQL, Spark SQL) across platforms such as Teradata and Databricks Unity Catalog.
• Implement and manage CI/CD pipelines to seamlessly deploy code artifacts to AWS and Databricks.
• Orchestrate and monitor Databricks jobs using Databricks Workflows, proactively troubleshooting and resolving issues.
• Contribute actively to Agile ceremonies, including sprint planning, grooming, daily stand-ups, demos, and retrospectives, fostering a culture of continuous improvement.

Requirements

• 8+ years of related hands-on experience Spark, PySpark, Shell scripting, Teradata, and Databricks.
• Proven ability to write complex and efficient SQL queries and stored procedures.
• Solid experience implementing data lake or data warehouse solutions on Databricks.
• Proficiency with Agile methodologies and DevOps tools such as Git, Jenkins, and Artifactory.
• Experience with Unix/Linux shell scripting (KSH) and basic Unix server administration.
• Familiarity with job scheduling tools like CA7 Enterprise Scheduler.
• Hands-on experience with AWS services including S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch.
• Expertise in Databricks components such as Delta Lake, Notebooks, Pipelines, cluster management, and cloud integration (Azure/AWS).
• Proficiency with collaboration tools like Jira and Confluence.
• Demonstrated creativity, foresight, and sound judgment in planning and delivering technical solutions.

Tech Stack

• AWS
• Azure
• Cloud
• EC2
• ETL
• Jenkins
• Linux
• PySpark
• Shell Scripting
• Spark
• SQL
• Unity
• Unix

Benefits

• Starting on day one of your employment, you’ll be offered several health-related benefits including medical, vision, dental, and well-being and behavioral health programs.
• 401(k)
• Company paid life insurance
• Tuition reimbursement
• Minimum of 18 days of paid time off per year
• Paid holidays