Systems Engineer III - Splunk / Cribl

Provide engineering and devops support of the CoLo migration with emphases on infrastructure supporting the Splunk and Cribl product lines. 1. Splunk Architecture & Administration Core Competencies: Design and maintain distributed Splunk deployments (search heads, indexers, forwarders, deployers) Manage indexer clustering and search head clustering for high availability Configure data inputs, parsing, and index management Implement role-based access control (RBAC) and authentication integration Performance tuning and capacity planning Data Onboarding: o Design and implement data onboarding strategies for diverse data sources o Create and maintain props.conf and transforms.conf for data parsing and routing o Develop source type definitions and field extractions o Configure input specifications and monitor data quality post-onboarding o Establish data retention policies and index lifecycle management Splunk HTTP Event Collector (HEC): o Configure and manage HEC endpoints for REST API-based data ingestion o Implement HEC tokens with appropriate permissions and index routing o Troubleshoot HEC connectivity, authentication, and data formatting issues o Scale HEC deployments for high-volume event ingestion o Integrate cloud-native applications and serverless functions with HEC Splunk DB Connect: o Install, configure, and maintain DB Connect app across search heads o Create database connections and manage JDBC drivers for various database types o Design and schedule database inputs (rising column, batch, and tail inputs) o Optimize SQL queries for performance and minimize database load o Configure database identity management and credential security o Troubleshoot connection issues, query timeouts, and data ingestion gaps Relevance: Essential for maintaining platform health, scalability, ensuring data availability across the enterprise, and enabling seamless integration of diverse data sources into the Splunk ecosystem 2. AWS Infrastructure & Services Core Competencies: Deploy and manage EC2 instances for Splunk components with proper sizing Configure VPCs, security groups, NACLs, and networking for secure Splunk communication Implement EBS storage optimization and snapshot strategies for Splunk data Leverage S3 for SmartStore architecture and backup solutions Use AWS Systems Manager, CloudWatch, and Auto Scaling for monitoring and automation Relevance: Critical for cost-effective, secure, and resilient infrastructure supporting enterprise-scale log aggregation 3. Infrastructure as Code (IaC) & Automation Core Competencies: Terraform or CloudFormation for provisioning Splunk infrastructure Ansible, Puppet, or Chef for Splunk configuration management Python/Bash scripting for custom automation tasks CI/CD pipeline integration (Jenkins, GitLab CI, GitHub Actions) Version control with Git for infrastructure and configuration code Relevance: Enables repeatable deployments, reduces human error, and accelerates disaster recovery and scaling operations 4. Monitoring, Logging & Troubleshooting Core Competencies: Create Splunk monitoring dashboards and alerts for platform health Implement log forwarding strategies using universal/heavy forwarders Troubleshoot data ingestion issues, search performance, and cluster health Integrate AWS CloudWatch metrics with Splunk for unified monitoring Analyze Splunk internal logs (_internal, _introspection, _audit indexes) Relevance: Ensures platform reliability, rapid incident response, and proactive identification of issues before they impact users 5. Security & Compliance Core Competencies: Implement encryption in-transit (SSL/TLS) and at-rest for Splunk data Configure AWS IAM roles and policies following least-privilege principles Ensure compliance with standards (PCI-DSS, HIPAA, SOC 2) for log data Implement backup and disaster recovery procedures Secure API access and credential management (AWS Secrets Manager, HashiCorp Vault) Relevance: Protects sensitive log data, maintains audit trails, and ensures regulatory compliance in enterprise environments 6. Cribl Stream & Cribl Edge - Data Pipeline Optimization Cribl Stream (LogStream) Competencies: Deploy and manage Cribl Stream architecture (Leader nodes, Worker nodes, Worker groups) Configure data sources and destinations for multi-platform routing (Splunk, S3, other SIEMs) Design and implement pipelines for data transformation, enrichment, and reduction Create routes and filters to optimize data flow and reduce ingestion costs Implement data sampling, aggregation, and redaction for compliance and cost savings Configure event breakers, parsers, and field extractions within Cribl Manage Cribl packs for pre-built data optimization solutions Integrate Cribl Stream with Splunk HEC and S3 for hybrid storage strategies Monitor pipeline performance and troubleshoot data flow issues Implement GitOps workflows for Cribl configuration management Cribl Edge Competencies: Deploy and manage Cribl Edge fleets for distributed edge data collection Configure Edge nodes as lightweight agents replacing traditional forwarders Implement centralized management of Edge fleets through Cribl Cloud or Stream Leader Collect data from edge sources (logs, metrics, Windows events, syslog) Perform edge-side data processing to reduce bandwidth and central processing load Configure auto-discovery and dynamic data source management Manage Edge node updates, configuration versioning, and fleet-wide deployments Monitor Edge node health and connectivity across distributed environments Implement edge-to-cloud data routing strategies for hybrid architectures 7. Incident Management & Service Request Support Core Competencies: Incident Response: o Triage and respond to platform incidents following ITIL or similar frameworks o Diagnose and resolve P1/P2 incidents affecting Splunk availability or data ingestion o Perform root cause analysis (RCA) and create post-incident reports o Coordinate with cross-functional teams during major incidents o Implement corrective and preventive actions to reduce incident recurrence o Maintain on-call rotation and provide 24/7 platform support Service Request Management: o Process user access requests (account creation, role assignments, permission changes) o Handle data onboarding requests for new applications and data sources o Fulfill infrastructure change requests (index creation, retention policy updates, capacity expansion) o Coordinate app installations and updates on search heads o Provision and configure new forwarders, HEC tokens, or DB Connect inputs o Create custom dashboards and reports based on user requirements Ticket Management & Communication: o Utilize ticketing systems (ServiceNow, Jira Service Management, Remedy) o Document troubleshooting steps and resolution procedures o Maintain SLA compliance for incident response and service request fulfillment o Communicate effectively with stakeholders on status updates and timelines o Create and maintain knowledge base articles for common issues o Escalate complex issues to vendors (Splunk Support, AWS Support) when necessary Proactive Support: o Conduct health checks and performance reviews o Identify trending issues and implement preventive measures o Provide user training and guidance on Splunk best practices o Participate in change advisory board (CAB) meetings for platform changes Relevance: Ensures rapid resolution of platform issues, maintains high availability and user satisfaction, and provides structured support that aligns with enterprise IT service management practices. Essential for maintaining operational excellence and meeting business-critical SLAs. 8. Agile Methodology & Project Collaboration Core Competencies: Scrum Framework Experience: o Participate in sprint planning, daily stand-ups, sprint reviews, and retrospectives o Commit to sprint goals and deliver incremental value within 2-week sprint cycles o Collaborate with Scrum Master to remove impediments and optimize team velocity o Contribute to backlog refinement and story estimation sessions (story points, planning poker) o Demonstrate completed work during sprint reviews and incorporate feedback o Identify process improvements during retrospectives for continuous team enhancement Kanban Framework Experience: o Work within continuous flow model with WIP (Work in Progress) limits o Manage work items through defined workflow stages (To Do, In Progress, Review, Done) o Prioritize tasks dynamically based on business value and urgency o Monitor cycle time and lead time metrics for process optimization o Participate in Kanban board reviews and workflow refinement o Balance operational support work with project-based initiatives Story Creation & Management: o Write clear, concise user stories with acceptance criteria following "As a [user], I want [goal], so that [benefit]" format o Break down epics into manageable user stories and technical tasks o Define technical requirements, dependencies, and effort estimates o Update story status, track progress, and document blockers in real-time o Create technical debt and bug stories with appropriate prioritization o Maintain story traceability through completion and closure Product Owner Collaboration: o Participate in backlog grooming sessions to clarify requirements and priorities o Provide technical feasibility input and effort estimates for proposed features o Communicate constraints, risks, and technical dependencies proactively o Negotiate scope and timelines based on technical complexity and resource availability o Seek clarification on ambiguous requirements before implementation o Provide regular status updates on work progress and potential delivery impacts o Offer alternative technical solutions to meet business objectives o Present completed work demonstrations and gather stakeholder feedback Agile Tools & Practices: o Utilize project management tools (Jira) o Maintain transparency through accurate story updates and burndown tracking o Participate in capacity planning and release planning activities o Contribute to definition of done (DoD) and team working agreements o Practice iterative development with continuous integration and delivery Relevance: Enables effective collaboration in fast-paced development environments, ensures alignment with business priorities, and facilitates continuous delivery of value. Critical for balancing platform operations with strategic initiatives and maintaining transparent communication with stakeholders in modern DevOps organizations. Education: Bachelors Degree Certification: AWS Developer , AWS Solutions Architect