Job Opportunity: Software Engineer Digital Brain CI/CD Asset Deployment Automation

Greetings from Infyshine Inc.,

Job Title: Software Engineer - Digital Brain CI/CD Asset Deployment Automation

Location: Raritan, NJ

Duration: 6 Months

Role Summary

We are looking for a Software Engineer to build an end-to-end Digital Brain CI/CD asset deployment automation framework. The engineer will design and develop a multi-file asset deployment workflow where multiple passport files are ingested, validated, sorted into a dependency DAG, and executed in the correct deployment order using topological sorting.

Key Responsibilities

Passport File Ingestion and Validation

Ingest multiple passport files received for asset deployment or update.

Validate each passport file to ensure all required deployment data is present.

Scan the passport file format and register valid passport files for processing.

Generate validation reports in JSON format.

Dependency DAG and Execution Order

Analyze dependencies between assets across multiple passport files.

Build an asset dependency DAG based on upstream and downstream relationships.

Implement topological sorting to determine the safe asset deployment execution order.

Ensure downstream assets wait until all required upstream dependencies are successfully completed.

Handle success dependency, control flow, data/artifact flow, and loop execution per passport file.

Asset Deployment Loop

Build the full deployment loop for each passport file in DAG execution order.

Extract changes from the passport file by comparing with previously deployed assets.

Support change detection for new, changed, and existing assets.

Generate required DDL scripts and deployment logs.

Deploy or update DDL changes into Databricks and Unity Catalog.

Ensure each passport file completes the full deployment loop before dependent assets are triggered.

Downstream System Integration

Develop integrations with the following enterprise systems:

Databricks
Unity Catalog
Alation
SharePoint
Knowledge Graph platforms
Vector databases
Ontology or semantic model platforms
REST APIs and metadata services

Build payload generation logic for each downstream system, including:

Alation metadata payloads
SharePoint document payloads
Knowledge graph payloads
Vector index payloads
Ontology or semantic model payloads

Implement parallel downstream deployment where applicable, while ensuring dependency and success/failure rules are respected.

Post-DDL Deployment Processing

Extract changes to push to Alation only after successful DDL deployment.

Extract changes to push to SharePoint only after successful DDL deployment.

Extract changes to push to downstream systems only after successful DDL deployment.

Support downstream updates for:

Alation metadata update or upsert
SharePoint document and metadata update
Knowledge Graph ingest or graph update
Vector Database vector upsert and indexing
Optional Ontological Model update and relationship management

Failure Handling and Auditability

Stop the asset deployment loop if any required extraction or deployment step fails.

Prevent downstream deployment when upstream deployment is unsuccessful.

Generate deployment logs, payload files, validation evidence, and audit artifacts.

Maintain traceability of generated artifacts for every passport file and deployment step.

Required Skills

Strong software engineering experience using Python or a similar backend programming language.

Experience building configuration-driven automation using JSON and YAML.

Strong understanding of DAGs, dependency resolution, graph processing, and topological sorting.

Experience developing CI/CD or deployment automation workflows.

Hands-on experience with Databricks and Unity Catalog deployment automation.

Experience integrating with enterprise platforms using REST APIs and metadata services.

Ability to build payload generation logic for multiple downstream systems.

Strong knowledge of validation, logging, error handling, audit tracking, and deployment status management.

Experience working with Git-based source control and environment-based deployments.

Disclaimer:

This e-mail contains confidential information or information belonging to Infyshine Technologies and is intended solely for the Addresses. The unauthorized disclosure, use dissemination or copying (either whole or partial of this e-mail, or any information it contains), is prohibited. E-mail is susceptible to alteration and their integrity cannot be guaranteed. Infyshine Technologies shall not be liable of this e-mail if modified or falsified. If you are not the intended recipient of this e-mail, please delete it immediately from your system & notify the sender of the wrong delivery and mail the deletion.