Senior Site Reliability Engineer, Observability

Chainlink Labs is the industry-standard oracle platform bringing the capital markets onchain and powering the majority of decentralized finance (DeFi). As a Senior Site Reliability Engineer, you will help accelerate and enable other engineering teams by increasing self-service and decreasing cognitive load while ensuring the reliability, security, and performance of observability services.

Responsibilities:

• Build and orchestrate Modern OTEL-based Observability Platform
• Support multiple telemetry types, like metrics, logs and traces
• Define and support modern governance in observability and problems at scale
• Ensure reliability, security, and performance exceed our defined SLAs
• Work with engineers from across the company to help troubleshoot issues, deploy new products and services, and increase velocity while decreasing cognitive load
• Lead the design and deployment of monitoring/observability services to detect and alert the team of needed action
• Ingest, aggregate, transform, and utilize data from a multitude of sources in our real time data pipeline
• Oversee the availability, performance, and supportability of our observability infrastructure
• Create processes around alert response operations and support the team to ensure the reliable delivery of oracle data
• Make recommendations to ensure sufficient metrics are collected to create alerts with every new feature release
• Champion reliability and security by taking the time to do your work right the first time

Requirements:

• 7+ years of relevant professional experience. You probably have worked on a devops, infrastructure, SRE, and/or platform team before
• Ability to develop software outside of the scope of typical infrastructure requirements and configurations
• Experience programming in C, C++, Java, Python, Go, Perl, or Ruby
• Expert knowledge in all aspects of designing, developing, and managing large real-time systems
• Experience with monitoring and logging. You know how to export metrics using Prometheus, have built a Grafana dashboard or two, and have experience with a centralized logging solution like an ELK Stack, Splunk or Grafana Stack
• Experience with distributed systems and container orchestration. You have maintained or even built Kubernetes clusters before and feel comfortable deploying completely new services on them
• Strong communication skills. You can give and receive constructive feedback, and you do not shy away from planning meetings and code reviews
• Excitement for blockchain, Web 3.0, and similar decentralized technologies
• Experience running any infrastructure in the blockchain/web3 space
• Ability to scale systems sustainably through mechanisms like automation, and evolve systems by pushing for changes that improve reliability and velocity
• Experience working remotely in a distributed team
• A strong desire to grow and challenge yourself. We would expect you to constantly find ways to improve and automate services to reduce toil