Lead Resiliency Engineer

Role Overview

• Lead and mentor a team in implementing solutions to enhance overall resiliency.
• Help define, document, and champion comprehensive resiliency engineering principles and practices tailored to the organization’s environment.
• Participate in the selection and implementation of tools for monitoring, alerting, chaos engineering, and automated recovery.
• Guide the analysis of ITSM workflows (incident, change, problem) to identify and prioritize high-impact automation opportunities.
• Architect and direct the development of automation solutions leveraging ITSM tools and integrating with other enterprise systems.
• Ensure the deployment and integration of automated workflows, maintaining seamless data flow and effective reporting.
• Establish and monitor metrics to track the effectiveness of resiliency and automation initiatives.
• Collaborate with Enterprise DevOps, Integration Platform DevOps, and other teams to review CI/CD pipelines, identify bottlenecks, and drive improvements.
• Participate in the development of a comprehensive automation strategy encompassing build, test, security scanning, and deployment processes to improve system resiliency.
• Participate in defining and standardizing operating environments and infrastructure configurations, partnering with architecture teams to implement Infrastructure-as-Code (IaC) frameworks.
• Implement and manage automated testing and rollback mechanisms for infrastructure and application changes.
• Lead the design, development, enhancement, and maintenance of tools, systems, and software solutions to support resiliency objectives.
• Direct incident triage efforts, determining scope, urgency, and potential impact, and coordinate effective response and recovery.
• Lead technology evaluations and re-engineering activities to support strategic direction and continuous improvement.
• Transform business requirements into technical specifications, ensuring alignment with resiliency and automation goals.
• Foster cross-functional coordination and ensure a partnership-focused approach to balancing service priorities with business needs, managing risk-based exceptions as necessary.

Requirements

• Experience in designing fault-tolerant architectures, including automated failover, multi-region redundancy, and graceful degradation strategies enabled by Chaos engineering.
• Deep understanding of complex distributed systems, such as microservices orchestration, service meshes, and eventual consistency models.
• Demonstrated experience delivering customer-driven solutions, support, or service in high-availability environments.
• Extensive knowledge of software engineering architectures, system/software design, and system deployments.
• Proven experience across multiple IT domains, including development, testing, configuration, deployment, and monitoring.
• Strong understanding of infrastructure technologies and application development methodologies.
• Demonstrated experience in system administration activities (configuration, installations, patch management, server maintenance) and network management (firewalls, proxies, IP management, routing, DNS).
• Experience in the utilization and support of integration and communication protocols between applications, databases, and technology platforms.
• Solid foundation in building scalable frameworks and providing specifications for APIs that support enterprise fulfillment processes.
• Strong analytical and problem-solving skills, with the ability to diagnose and resolve complex technical issues.
• Proven ability to lead and mentor technical teams, driving continuous improvement and innovation in resiliency engineering.

Tech Stack

• Distributed Systems
• DNS
• Firewalls
• ITSM
• Microservices

Benefits

• comprehensive medical, dental, vision and wellbeing benefits
• competitive 401(k) contribution
• pension plan
• annual incentive
• 9 paid holidays
• paid time off program (23 days accrued annually for full-time employees)
• student loan repayment program
• paid-family leave