Lead Site Reliability Engineer

Lumen Technologies is a trusted network provider for the AI-powered world, connecting people and data through an expansive fiber network. They are seeking a Lead Site Reliability Engineer to focus on production support and performance optimization, ensuring the reliability and efficiency of systems while leveraging AI tools and collaborating with engineering teams.

Responsibilities:

• Implement AI systems and automations to assist during ongoing outages and triage potential ones. You will work with the development teams to ensure that they have all the data normally needed during an outage at their fingertips including preliminary analysis by AI
• Provide Tier 3 support for issues across portal services by troubleshooting and resolving technical issues in test and production environments
• Lead root cause analysis and post-mortem processes, incorporating AI-assisted analysis and pattern detection to ensure continuous improvement
• Monitor system performance and proactively identify bottlenecks or degradation using AI-driven observability and anomaly detection tools
• Implement tuning strategies across application layers, databases, and infrastructure
• Drive initiatives to improve latency, throughput, and resource utilization
• Deploy improved alerting for Lumen Connect in depth, focusing on outside in but including early indicators for fulfilment and other areas. Combining traditional monitoring with AI-based anomaly detection and noise reduction
• Proactively monitor the errors and performance on Lumen Connect. Implement rules to detect deviations, implement improvements together with the teams
• Design and maintain dashboards, alerts, and metrics using tools like Datadog, AppInsights, CloudWatch, or similar
• Develop and maintain automation scripts and tools for deployment, scaling, and recovery
• Develop and maintain automation scripts and tools for deployment, scaling, and recovery, leveraging AI-assisted code generation and validation tools
• Use Terraform, or similar IaC tools to manage AWS resources
• Perform an in-depth analysis of the overall system and its dependencies, implementing techniques to increase the global availability, reduce the reliance on unstable dependencies and guide ecosystem improvements
• Champion SRE principles such as SLIs, SLOs, and error budgets
• Advocate for resilient architecture and fault-tolerant design patterns, incorporating AI-assisted design reviews and architecture evaluation
• Work closely with software engineers, DevOps, and product teams to align reliability goals
• Document processes, runbooks, and best practices for knowledge sharing
• Provide mentorship and guidance on reliability and operational excellence

Requirements:

• 8+ years overall professional experience in SRE, DevOps, or infrastructure engineering roles
• Experience with Terraform, or similar IaC tools to manage Cloud resources
• Proficiency in scripting languages (Python, Bash, etc.) and automation frameworks
• Experience with CI/CD pipelines and tools like GitHub Actions, Jenkins or GitLab CI
• Solid understanding of monitoring and logging tools (e.g., CloudWatch, ELK, Datadog)
• Familiarity with containerization and orchestration (Docker, Kubernetes)
• Excellent AI and problem-solving skills, and a proactive mindset
• Experience in AWS services (EC2, CloudFront, EKS, RDS, S3, etc.)
• Certifications in AWS or related technologies are a plus
• Experience of application development using Java Microservices and Spring Boot framework
• Experience with Agile/SCRUM Methodologies and development practices