Develop and maintain data pipelines and notebooks using Azure Databricks under senior guidance.
Assist in monitoring cloud spend and resource utilization using Unravel and Azure tools, identify cost anomalies, investigate usage spikes, and collaborate with senior team members to implement cost optimization solutions and provide visibility to leadership.
Monitor Databricks jobs and clusters, escalating performance or cost issues as needed.
Collaborate with data engineers, analysts, and cloud architects on data and cost optimization initiatives.
Troubleshoot data pipeline failures and performance issues.
Follow data governance, security, and coding best practices.
Document solutions, processes, and lessons learned.
Continuously learn and apply new programming skills and technologies to meet evolving project needs.
Handle assignments with increasing complexity and scope.
Develop an understanding of systems, technology, and organizational structure within the company.
Requirements
Bachelor’s degree in computer science, Software Engineering, or a related field
0 to 2 years of relevant experience
Experience with Python and SQL
Hands-on experience, coursework, internship, or certification in a major cloud platform (Azure preferred; AWS or GCP also considered)
Basic understanding or hands-on exposure to Apache Spark/Azure Databricks
Understanding of basic software development practices (e.g., version control, code reviews)
Strong willingness to learn cloud data engineering and cost optimization concepts
Highly motivated, self-starting team player with a willingness to learn and take on new challenges
Comfortable working in a fast-paced, dynamic environment
Demonstrated ability to learn new technologies quickly and adapt to evolving cloud, data, and AI-driven platforms
Strong problem-solving skills and ability to work effectively in a team
Excellent communication skills and attention to detail
Eager to adapt to new challenges and technologies
Openness to feedback, coaching, and collaboration with peers and senior team members
Tech Stack
Apache
AWS
Azure
Cloud
Google Cloud Platform
Python
Spark
SQL
Benefits
Health benefits effective day 1
Paid time off, holidays, volunteer time and jury duty pay
Recognition pay
401(k) retirement savings plan with employer match