Site Reliability Engineer - OpenTelemetry

Intuitive.ai is an AI-first engineering company delivering measurable business outcomes across various domains. They are seeking a Senior Site Reliability Engineer with expertise in OpenTelemetry to join their centralized observability program, focusing on consolidating telemetry data and enhancing platform reliability.

Responsibilities:

• Design, build and continuously modify OpenTelemetry collector pipelines — receivers, processors, exporters and connectors across on-premises, AWS and GCP source classes
• Engineer the ingest economics of the platform: sampling and deduplication strategy, attribute pruning, cardinality control and log-to-metric conversion, reducing volume without losing operational signal
• Implement enrichment and redaction at the point of collection, including hashing and masking of sensitive fields and tagging telemetry with team, application and environment identifiers
• Build and evolve the dashboard layer on ClickHouse — templatized, parameterized dashboards per domain, persona-based views for distinct audiences, and business-impact dashboards linking user-facing symptoms to underlying infrastructure causes
• Author and optimize the analytical SQL behind those dashboards, including materialized views and projections for common query patterns
• Design the alerting baseline: threshold definition, alert quality and noise reduction, and routing into ITSM (ServiceNow) for automated incident creation
• Establish SLI / SLO frameworks with error budget policies for in-scope services, and drive the shift from siloed tool-based monitoring to unified, proactive observability
• Operate the collector gateway workloads deployed as Kubernetes pods on AWS and GCP, and as virtual machines on-premises, including manifest application, scaling and troubleshooting
• Support parity validation during migration — dual-destination routing so telemetry lands in both the legacy and target platforms until confidence is established for cutover
• Work directly with application teams to onboard their services and signals onto the platform, guiding instrumentation choices and service naming, tagging and ownership conventions normalization
• Troubleshoot across domain boundaries — networking, firewalls, load balancing, cloud services, database and application layers — isolating the failing domain and routing issues to the right owner
• Document pipeline patterns, dashboard standards and operational runbooks to support scaling beyond the initial rollout

Requirements:

• Deep, hands-on OpenTelemetry expertise — collector architecture, OTLP, semantic conventions, and the design and modification of production pipelines across receivers, processors, exporters and connectors. Able to explain why a pipeline is shaped a certain way, including processor ordering and its consequences, not only replicate an existing configuration
• Telemetry pipeline and cost engineering — sampling and deduplication strategy, filtering, enrichment, attribute manipulation, cardinality management, and multi-destination routing, with a clear grasp of the trade-off between ingest cost and observability fidelity
• Instrumentation experience — auto-instrumentation and SDK-based instrumentation across common runtimes such as Java, .NET, Node, Python and Go
• MELT mastery — fluency across metrics, events, logs and traces, including collection, and enrichment, and sound judgment on which signal type answers which operational question
• SRE depth — SLI / SLO design and error budget policy, alerting design and noise reduction, event correlation, and incident triage from user-facing symptom through to probable root cause
• Strong analytical SQL — aggregations, joins, window functions, percentile calculation and time bucketing against large datasets, written directly rather than through a visual query builder or a proprietary vendor query language
• Proven dashboarding capability on any modern observability stack — Grafana, Kibana, Datadog, New Relic, Dynatrace, Splunk or equivalent — building reusable, parameterized dashboards from raw telemetry rather than one-off views
• Holistic, cross-domain perspective — a working viewpoint spanning application, network, security and database observability, sufficient to pinpoint and correctly route problems across those boundaries
• Strong stakeholder communication — effective embedded within a client platform team, guiding application owners of varying technical maturity through onboarding, and operating across distributed, multi-timezone teams
• ClickHouse and ClickStack / HyperDX — query performance tuning, materialized views, projections, TTL policies, and row-level security implemented through data enrichment and metadata-driven access mapping
• Collector control plane tooling — Bindplane and BDOT, or comparable fleet-management tooling for agent and gateway configuration, versioning and staged rollout
• Cloud familiarity across AWS and GCP — cloud-native telemetry paths including CloudWatch, Kinesis Data Firehose, Pub/Sub, Cloud Logging sinks, VPC flow logs and audit logs, together with IAM, service accounts and secure service integration. Familiarity is expected, deep architecture specialization is not
• Kubernetes operational capability — deploying and operating collector workloads on EKS and GKE, applying and troubleshooting manifest files, DaemonSets, resource sizing and pod scaling. Helm chart authoring is not required, as collector manifests are generated by the platform
• GitOps and configuration-as-code — versioning, approval and promotion workflows for pipeline YAML, applied via platform REST endpoints. This is a later-phase requirement on this program
• Platform migration experience — moving from a proprietary APM such as AppDynamics, or a log platform such as Splunk, onto OpenTelemetry, including correlation-context migration and parity validation
• Regulated-environment exposure — healthcare, PHI / PII handling, or comparable compliance-sensitive telemetry work