Databricks is a leading data and AI company that empowers organizations worldwide to unify and democratize data, analytics, and AI. The Staff Security Detection Engineer will play a critical role in safeguarding the company's products and infrastructure by designing and implementing scalable intrusion detection solutions using advanced machine learning techniques and collaborating with cross-functional teams.
Responsibilities:
- Design and implement advanced detection strategies by deeply understanding and analyzing new or unknown log sources, schemas, and raw data
- Collaborate with cross-functional teams, including product and data engineering teams, to build efficient log ingestion pipelines and support large-scale data analytics
- Engineer and deploy detection solutions on Databricks using Spark, Python, and other cutting-edge technologies with a strong emphasis on clean code, rigorous testing, and comprehensive documentation
- Develop Rule-based and/or ML-based intrusion detection models and integrate them with Databricks' platform, ensuring high accuracy and minimal false positives
- Partner with Incident Response teams to perform threat hunting and to provide detailed logging, alerts, and playbooks, empowering proactive threat detection and response
- Influence the development of long-term technology strategies and roadmaps for detection engineering, ensuring alignment with broader business and security goals
- Represent Databricks at security and engineering conferences, presenting novel detection approaches and thought leadership within the security community
Requirements:
- 10+ years of relevant experience or advanced degree + 7 years of experience, with a focus on security detection engineering
- 6+ years of software engineering experience, with 4+ years specifically in security-related engineering, particularly in detection engineering
- Expertise in securing and operating at least one major cloud environment (AWS, Azure, GCP)
- Strong technical proficiency in key areas such as network security, cloud security, application/log analysis, and endpoint security
- Proven experience in Python, Git/GitHub, and CI/CD automation
- Familiarity with distributed computing environments (e.g., Pyspark), SQL, data analysis tools, and machine learning
- A strong passion for continuous learning and staying updated on evolving attack techniques and defense strategies
- Excellent communication skills, with the ability to collaborate effectively across teams and present complex ideas clearly to stakeholders at all levels
- A leadership mindset with the ability to mentor peers, drive strategic initiatives, and influence the organization's security direction
- terraform knowledge is a plus