Senior Data Engineer – Hadoop

Bright Vision Technologies is a technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States. The Senior Data Engineer - Hadoop will design, build, and operate large-scale data processing pipelines and analytics platforms using Hadoop and related big-data ecosystems. The role focuses on ingesting, transforming, and analyzing structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads.

Responsibilities:

• Design, build, and operate large-scale data processing pipelines and analytics platforms on Hadoop and related big-data ecosystems
• Ingesting, transforming, and analyzing massive volumes of structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads

Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related technical discipline
• Five or more years of professional experience designing and operating big-data pipelines on Hadoop
• Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem
• Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink
• Strong SQL skills and experience working with both relational and NoSQL data stores
• Experience with workflow orchestration tools such as Airflow or Oozie
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance
• Strong scripting skills in Python or Shell
• Excellent troubleshooting, debugging, and documentation skills
• Experience operating Hadoop on cloud platforms such as AWS EMR, Azure HDInsight, or Databricks
• Familiarity with modern lakehouse formats (Delta, Iceberg, Hudi)
• Exposure to data governance tooling such as Apache Atlas or Collibra
• Experience with Kubernetes-based data platforms (Spark-on-K8s, Trino)
• Hands-on experience with CI/CD and infrastructure-as-code in data engineering workflows