Senior Big Data Engineer

Net2Source is a Global Workforce Solutions Company headquartered at NJ, USA, and they are seeking a Senior Big Data Engineer for one of their clients. The role involves owning data pipelines, building foundational and derivative data products, and managing various data ingestion and processing frameworks.

Responsibilities:

• Owns the data pipelines that move data from operational source systems onto the data platform extraction, ingestion, transformation, orchestration, and the day-to-day operational health of those pipelines
• Also plays a role as a hands-on builder of Foundational Data Products (raw record-of-truth) and potentially Derivative Data Products (composed from upstream products)
• Pipeline build & operations: design, develop, deploy, monitor, and remediate batch and streaming pipelines that land source data on the platform; manage backfills, replays, late-arriving data, schema drift, and SLA breaches
• Ingestion patterns: full-load, incremental, change-data-capture (Debezium, Fivetran, Qlik Replicate, GoldenGate), event-driven ingest, API and file-based intake
• Pipeline frameworks: dbt, Apache Spark, Apache Beam, Airflow, Dagster, Prefect
• Streaming: Kafka, Kinesis, Flink, Spark Structured Streaming
• Data product packaging: schema contracts (Avro/Protobuf/JSON Schema), versioning, SLAS/SLOs, data contracts, output-port design (SQL, file, API, event)
• Storage formats: loeberg, Delta Lake, Hudi (open table formats are now the mesh default)
• Quality & observability: Great Expectations, Soda, Monte Carlo, dbt tests, data lineage (OpenLineage)
• CI/CD for data: GitOps pipelines, unit + integration tests, environment promotion
• Al-adjacent vector store ingestion (pgvector, Pinecone), feature stores (Feast, Tecton), RAG-ready chunking and embedding pipelines
• Mesh-specific: knowing when to build a derivative product vs. extending an existing one; consuming upstream products through governed input ports rather than reaching into source systems

Requirements:

• SQL & Python (table stakes)
• Scala/Java for high-throughput streaming
• Pipeline build & operations: design, develop, deploy, monitor, and remediate batch and streaming pipelines that land source data on the platform; manage backfills, replays, late-arriving data, schema drift, and SLA breaches
• Ingestion patterns: full-load, incremental, change-data-capture (Debezium, Fivetran, Qlik Replicate, GoldenGate), event-driven ingest, API and file-based intake
• Pipeline frameworks: dbt, Apache Spark, Apache Beam, Airflow, Dagster, Prefect
• Streaming: Kafka, Kinesis, Flink, Spark Structured Streaming
• Data product packaging: schema contracts (Avro/Protobuf/JSON Schema), versioning, SLAS/SLOs, data contracts, output-port design (SQL, file, API, event)
• Storage formats: loeberg, Delta Lake, Hudi (open table formats are now the mesh default)
• Quality & observability: Great Expectations, Soda, Monte Carlo, dbt tests, data lineage (OpenLineage)
• CI/CD for data: GitOps pipelines, unit + integration tests, environment promotion
• AI-adjacent vector store ingestion (pgvector, Pinecone), feature stores (Feast, Tecton), RAG-ready chunking and embedding pipelines
• Mesh-specific: knowing when to build a derivative product vs. extending an existing one; consuming upstream products through governed input ports rather than reaching into source systems