Key skills
Apache NiFiNiFi processorsNiFi controller servicesNiFi templatesNiFi reporting tasksData integrationETLJSONXMLCSVAvroParquetBatch data processingReal-time data processingIoT data processingIndustrial sensorsPLCsSCADA systemsIndustrial communication protocolsMQTTOPC-UAModbusDNP3Ethernet/IPReal-time telemetry data collectionAnomaly detectionData filteringData enrichmentICS environmentsHigh-volume data streamsLatency managementReliability managementIndustrial IoT platformsTime-series databasesICS securityNetwork segmentationSecure communication TLSRole-based access controlGroovy scriptingPython scriptingShell scriptingJava programmingKafkaHadoopSparkMessaging systemsStreaming data pipelinesLinuxUnixNetworking conceptsSSL TLSAuthentication mechanismsNiFi monitoringNiFi troubleshootingNiFi logsNiFi provenance dataNiFi metricsGitVersion controlCI/CD pipelinesPythonJavaShellAnalyticsVersion ControlCI/CDCommunicationAccount Management
About this role
OP is a technology consulting and solutions company that helps harness the power of technology for maximum impact. They are seeking an Apache NiFi Engineer to design, deploy, and maintain data pipelines for real-time analytics and industrial IoT solutions.
Responsibilities:
- Join our team as an Apache NiFi Engineer and play a vital role in designing, deploying, and maintaining high-performance data pipelines that power real-time analytics and industrial IoT solutions
- You'll be responsible for managing, securing, and troubleshooting NiFi clusters, optimizing data flows, and ensuring system reliability across complex environments involving IoT devices, SCADA systems, and big data platforms
- With your expertise in data integration, industrial protocols, scripting, and system monitoring, you'll help deliver scalable, secure, and efficient data solutions that drive innovation and operational excellence
- Participate in OP monthly team meetings and participate in team-building efforts
- Contribute to OP technical discussions, peer reviews, etc
- Contribute content and collaborate via the OP-Wiki/Knowledge Base
- Provide status reports to OP Account Management as requested
Requirements:
- Strong understanding of Apache NiFi
- Experience designing, developing, and deploying data flows using NiFi
- Knowledge of NiFi processors, controller services, templates, and reporting tasks
- Ability to configure and optimize NiFi for performance and reliability
- Experience with data ingestion, transformation, routing, and system integration
- Familiarity with common data formats (JSON, XML, CSV, Avro, Parquet)
- Understanding of batch and real-time data processing concepts
- Experience ingesting and processing IoT data from industrial sensors, PLCs (Programmable Logic Controllers), SCADA systems, and other ICS devices
- Familiarity with industrial communication protocols such as MQTT, OPC-UA, Modbus, DNP3, and Ethernet/IP
- Ability to design NiFi flows for real-time telemetry data collection, anomaly detection, filtering, enrichment, and routing within ICS environments
- Handling high-volume, time-sensitive data streams with strict latency and reliability requirements
- Integration with industrial IoT platforms, historians, time-series databases, and analytics/monitoring tools
- Understanding of ICS-specific security concerns, including network segmentation, secure communication (TLS), and role-based access control
- Proficiency in scripting languages such as Groovy, Python, or Shell for custom processors or extensions
- Knowledge of Java is a plus for developing custom NiFi processors
- Experience with Kafka, Hadoop, Spark, or other big data ecosystems is beneficial
- Understanding of messaging systems and streaming data pipelines
- Familiarity with Linux/Unix environments where NiFi typically runs
- Understanding of networking concepts, security (SSL/TLS), and authentication mechanisms
- Ability to monitor NiFi flows, troubleshoot issues, and optimize resource usage
- Experience with NiFi logs, provenance data, and metrics
- Experience with Git or other version control systems
- Familiarity with CI/CD pipelines for deploying NiFi flows
- Strong problem-solving and analytical skills
- Good communication skills to collaborate with data engineers, architects, and stakeholders