Payrate: $92.00 - $98.00/hr.

Summary:
We are seeking a hands-on Data Pipeline Engineer to build and operate Python-based workflows that transform large document and text datasets into high-quality data for AI model training. This role combines OCR processing, data cleaning and validation, scalable pipeline development, and Azure storage management.

Responsibilities

Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
Run and monitor large-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms.
Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats.
Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage; verify transfer completeness and data integrity.
Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
Optimize processing throughput, memory usage, storage consumption, and cloud costs using appropriate batch, parallel, or distributed processing approaches.
Track dataset statistics, processing coverage, quality metrics, and exceptions; produce clear operational reports.
Collaborate with applied scientists and model-training teams to translate data requirements into reproducible processing workflows.
Maintain tested, documented, version-controlled code and follow established data access, security, and governance requirements.

Qualifications

Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities.
Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
Hands-on experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
Working knowledge of SQL, Git, Linux environments, and command-line tools.
Strong troubleshooting skills, attention to data quality, and ability to communicate technical issues clearly.

Skills

Experience with OCR pipelines, scanned documents, multilingual text, or document-layout processing.
Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools.
Familiarity with Hugging Face Datasets, tokenization, near-duplicate detection, and training/evaluation data separation.
Experience with dataset lineage, reproducibility, containerized environments, and CI/CD.

Pay Transparency: The typical base pay for this role across the U.S. is: $92.00 - $98.00 /hour. Non-exempt positions are eligible for overtime at a rate of 1.5 times the base hourly rate for all hours worked in excess of 40 in a work week, or as required by state or local law. Final offer amounts, within the base pay set forth above, are determined by factors including your relevant skills, education and experience. Full-time employees are eligible to select from different benefits packages. Packages may include medical, dental, and vision benefits, health savings accounts with qualified medical plan enrollment, 10 paid days off, 3 days paid bereavement leave, 401(k) plan participation with employer match, life and disability insurance, commuter benefits, dependent care flexible spending account, accident insurance, critical illness insurance, hospital indemnity insurance, accommodations and reimbursement for work travel, and discretionary performance or recognition bonus. Sick leave and mobile phone reimbursement provided based on state or local law.

Consent to Communication and Use of AI Technology: By submitting your application for this position and providing your email address(es) and/or phone number(s), you consent to receive text (SMS), email, and/or voice communication whether automated (including auto telephone dialing systems or automatic text messaging systems), pre-recorded, AI-assisted, or individually initiated from Aditi Consulting, our agents, representatives, or affiliates at the phone number and/or email address you have provided. These communications may include information about potential opportunities and information. Message and data rates may apply. Message frequency may vary.
You represent and warrant that the email address(es) and/or telephone number(s) you provided to us belong to you and that you are permitted to receive calls, text (SMS) messages, and/or emails at these contacts. You also acknowledge and agree to Aditi Consulting LLC’s use of AI technology during the sourcing process, including calls from an AI Voice Recruiter. AI is used solely to gather data and does not replace human-based decision-making in employment decisions.  Calls may be recorded.

Consent is not a condition of purchasing any property, goods, or services. You may revoke your consent at any time by replying “STOP” to messages or by contacting  .
For information about our collection, use, and disclosure of applicant's personal information as well as applicants' rights over their personal information, please see our .

#AditiConsulting
# 26 - 07199

AI Data Engineer

Key skills

About this role