Elastic Pipeline Engineer/Architect

IT Minds LLC is seeking an Elastic Pipeline Engineer to build an automated redaction and evidence pipeline within the Elastic platform. The role involves collaborating with architects and stakeholders to enhance the evidence lifecycle, ensuring compliance and security through effective data management and redaction techniques.

Responsibilities:

• Participate in discovery sessions alongside the Lead Architect and engineering and compliance stakeholders to map the current evidence lifecycle, identify manual touchpoints and custody gaps, and confirm the redaction ruleset
• Review the existing Elastic deployment to understand current index structure, ingestion patterns, access control configuration, and data retention policies, and evaluate candidate integration points at the Elastic ingest layer
• Develop and configure the automated redaction and evidence pipeline within the Elastic ingest layer in accordance with the approved solution design
• Implement field-level redaction rules dropping, masking, or hashing traceability fields per the data classification policy and unit-test them against representative pipeline output samples across all classified field types
• Validate that tamper-evidence metadata is correctly stamped at execution and preserved accurately within Elasticsearch to maintain an immutable chain of custody
• Configure Elasticsearch index templates, field mappings, and Index Lifecycle Management (ILM) policies aligned to audit and data retention requirements
• Configure auditor-facing search and retrieval views, implementing the role-based access model that distinguishes auditor views from operational engineering views
• Conduct integration and acceptance testing for evidence retrieval and data redaction against agreed acceptance criteria, documenting test results and remediation actions prior to handover
• Support UAT, capturing and resolving defects identified in the auditor retrieval workflow prior to engagement closure
• Contribute to pipeline architecture documentation, operational runbooks, and knowledge transfer sessions with the engineering and operations teams
• Participate in agile ceremonies daily stand-ups, sprint demos, and retrospectives

Requirements:

• Elastic Stack Elasticsearch, ingest pipelines, index templates & ILM 5
• Log pipeline / data transformation tooling (Logstash, ingest processors, Beats) 5
• Python for data / integration automation 4
• Data pipeline / ETL-style development 3
• Working with structured & semi-structured data (JSON, schemas, field mappings) 3
• Git / GitOps workflow 3
• Data redaction, masking, or hashing techniques for sensitive fields 2
• Role-based access control & search/retrieval configuration in Elastic (Kibana) 2
• Working knowledge of data privacy / PII handling & data classification 2
• CI/CD pipeline development (TeamCity, GitLab CI, Jenkins, or equivalent) 2
• YAML / JSON data modelling 2
• Linux / Bash 2
• Prior experience delivering tamper-evident or chain-of-custody logging solutions in regulated environments
• Experience with Elastic security features field- and document-level security, role mapping, and audit logging
• Familiarity with CI/CD-driven change pipelines and pre-check / post-check validation outputs
• Experience integrating evidence or audit workflows for compliance and regulatory review
• Experience in regulated environments (financial services, GDPR)
• Client-facing professional services delivery experience