Principal Engineer Software, Dev & Production Infrastructure (Chronosphere)

Palo Alto Networks is a leader in cybersecurity, dedicated to protecting the digital way of life through innovative technology. They are seeking a Principal Engineer for their Chronosphere platform, focusing on optimizing and ensuring the reliability of their global production infrastructure and enhancing developer velocity through advanced tooling and systems architecture.

Responsibilities:

• Solve complex distributed systems problems at the largest scales in the world
• Scale our production infrastructure and architecture globally using Kubernetes, GCP, and AWS
• Use Go to build and operate highly scalable, resilient backend services
• Develop the internal tools necessary to monitor, alert, and optimize our live production footprint
• Architect & Build: Design high-scale developer tooling, dynamic testing environments, and CI/CD pipelines in a 100% modern, containerized microservices ecosystem
• Infrastructure as Code (IaC): Treat infrastructure as a first-class citizen by defining and managing entire ephemeral environments using declarative IaC (Terraform)
• Drive Systemic Quality: Identify and eliminate systemic bottlenecks across the development lifecycle through architectural changes, advanced tooling, and near-real-time telemetry processing
• Own massive technical initiatives from inception to delivery, balancing feature velocity with long-term technical debt
• Define platform standards and reference architectures that span a 1–3 year horizon
• Act as the "glue" across the organization, consulting on infrastructure best practices and up-leveling the team through dedicated mentorship

Requirements:

• 8–10+ years of relevant experience in high-scale infrastructure, production engineering, systems architecture, or developer platform roles
• 7 + years of experience in at least one backend language (e.g., Go, Python, Java, C++, C#, or Rust). We value modular, testable code over specific syntax knowledge, though Go is our primary language
• 5 + years of deep expertise building and debugging systems that deal with CAP theorem trade-offs, eventual consistency, and distributed tracing. Robust experience working with AWS or GCP and navigating Kubernetes clusters
• Deep knowledge of Linux internals, process management, resource isolation, and networking protocols (OSI model, load balancing, service meshes)
• A proven track record of estimating complex work effectively, debugging distributed architectures via logs/traces, and proactively identifying design edge cases
• Active engagement or experience with open-source communities
• Experience or active interest in using AI coding assistants (like Cursor or Claude) to accelerate engineering velocity and automate boilerplate tasks