Senior Data Engineer - NiFi

Key Responsibilities:

NiFi Platform Assessment & Optimization

• Review existing Cloudera NiFi flows and architectures for efficiency, scalability, and maintainability.
• Assess current use cases to ensure NiFi is being applied appropriately and effectively.
• Identify opportunities to simplify flow designs, reduce operational overhead, and improve performance.
• Provide recommendations on best practices, standards, and reusable patterns.

Data Integration & Pipeline Support

• Support and enhance ETL and ELT pipelines built on NiFi.
• Integrate data from diverse sources including databases, APIs, filesystems, streaming platforms, and cloud storage.
• Troubleshoot data flow failures, latency issues, and throughput constraints.
• Implement robust error handling, monitoring, and recovery mechanisms.

Python & Custom Development

• Use Python for transformations, scripting, automation, and custom processor logic (where applicable).
• Support integration between NiFi and external systems using custom scripts or utilities.

Governance, Security & Operations

• Apply best practices for secure data movement, authentication, authorization, and encryption.
• Assist with flow versioning, deployment processes, and operational runbooks.
• Collaborate with internal teams to ensure platform changes align with operational and compliance requirements.

Advisory & Collaboration

• Act as a technical advisor to internal data and platform teams.
• Clearly document findings, recommendations, and implemented improvements.
• Provide knowledge transfer and guidance to ensure longterm sustainability.

Required Experience & Skills

• Proven, handson experience with Cloudera NiFi in production environments.
• Strong knowledge of ETL/ELT concepts and modern data pipeline architectures.
• Proficiency in Python for data processing, scripting, and automation.
• Experience working with Linuxbased systems and distributed environments.
• Ability to analyze existing implementations and deliver actionable optimization recommendations.

Preferred Experience

• Experience within the Cloudera ecosystem (e.g., Kafka, HDFS, Hive, Iceberg).
• Familiarity with CI/CD practices for data pipelines.
• Experience with monitoring and tuning distributed data platforms.
• Exposure to cloud and hybrid data architectures.

Engagement Expectations

• Deliverablesfocused, handson consulting role.
• Ability to work independently while collaborating with internal stakeholders.
• Clear written and verbal communication of technical findings and recommendations.