Key skills

Data ArchitectureData ModelingData Pipeline DesignSQLPythonApache SparkApache FlinkApache KafkaApache AirflowDagsterPrefectdbtDuckDBInfrastructure as Code (IaC)CI/CDKubernetesCloud InfrastructureLarge Language Models (LLMs)Retrieval-Augmented Generation (RAG)Written and Verbal CommunicationAIMachine LearningLLMLarge Language ModelsData LakeVersion ControlCachingDecision Making

About this role

Catalyst Labs is a leading talent agency specializing in Applied AI, Machine Learning, and Data Science, recruiting for a venture-backed advertising technology startup. The Tech Lead, Data & Inference Engineer will lead the design and scaling of an end-to-end data platform, build reliable batch and streaming pipelines, operate LLM-powered inference workflows, and guide architecture, integrations, data quality, and engineering mentorship.

Responsibilities:

Lead the design, development and scaling of an end to end data platform from ingestion to insights, ensuring that data is fast, reliable and ready for business use
Build and maintain scalable batch and streaming pipelines, transforming diverse data sources and third party application programming interfaces into trusted and low latency systems
Take full ownership of reliability, cost and service level objectives. This includes achieving ninety nine point nine percent uptime, maintaining minutes level latency and optimizing cost per terabyte. Conduct root cause analysis and provide long lasting solutions
Operate inference pipelines that enhance and enrich data. This includes enrichment, scoring and quality assurance using large language models and retrieval augmented generation. Manage version control, caching and evaluation loops
Work across teams to deliver data as a product through the creation of clear data contracts, ownership models, lifecycle processes and usage based decision making
Guide architectural decisions across the data lake and the entire pipeline stack. Document lineage, trade offs and reversibility while making practical decisions on whether to build internally or buy externally
Scale integration with application programming interfaces and internal services while ensuring data consistency, high data quality and support for both real time and batch oriented use cases
Mentor engineers, review code and raise the overall technical standard across teams. Promote data driven best practices throughout the organization