Senior Data Engineer

SMX Services & Consulting, Inc. is seeking a Senior Data Engineer to support the Technology Solutions Division of the SBA Office of Inspector General. The Senior Data Engineer will design, implement, maintain, secure, document, and optimize an enterprise data platform within the SBA Microsoft Azure environment.

Responsibilities:

• Provide senior-level guidance on data-engineering methods, code-first development, cloud architecture, and modern pipeline design
• Design, implement, and maintain a secure, stable, scalable, flexible, and efficient Azure data architecture
• Maintain data-platform assets, code, configurations, pipelines, and related artifacts in approved source-control repositories
• Design, implement, operate, and maintain ETL and ELT pipelines for Azure Synapse and Azure Machine Learning
• Support Azure Machine Learning SDK v1 and SDK v2 environments
• Review and improve existing data architecture, repositories, datasets, models, and pipelines
• Conduct periodic architecture and pipeline reviews to identify bottlenecks, deprecated dependencies, configuration issues, and architecture drift
• Establish pipeline quality controls, including logging, error handling, validation, reconciliation, alerting, and monitoring
• Implement source-control practices across data-engineering and analytics codebases
• Support iterative code development while protecting production stability
• Optimize the ingestion, transformation, processing, storage, and retrieval of structured, semi-structured, and unstructured datasets
• Work with SQL databases, APIs, files, data lakes, data warehouses, and modern columnar formats such as Parquet
• Develop self-service capabilities that allow SBA OIG analysts to query and export data for audits and investigations
• Coordinate with Senior Data Scientists to ensure the architecture supports statistical models, machine-learning algorithms, NLP workloads, and production analytical pipelines
• Develop standard operating procedures governing pipeline development, testing, validation, publishing, execution, monitoring, and maintenance
• Produce and maintain data dictionaries, entity-relationship diagrams, architecture diagrams, pipeline maps, and technical operating documentation
• Add new datasets, pipelines, services, and integrations through defined intake, testing, approval, and deployment processes
• Develop datasets supporting loan-fraud identification, federal audits, and criminal investigations
• Optimize frequently used database queries
• Develop and implement data models that improve database efficiency and usability
• Recommend architectural improvements that reduce Azure cloud costs
• Monitor pipelines to ensure timely processing and regular dataset updates
• Work with cloud-architecture personnel to improve methods of accessing and transporting data
• Support dashboards, reports, visualizations, and data storytelling
• Stay current with emerging artificial-intelligence tools relevant to data engineering
• Support approved exploratory automation and LLM-assisted capabilities

Requirements:

• Must be a U.S. citizen
• Possess a valid Social Security number
• Be eligible to undergo a Public Trust background investigation
• Receive a favorable Entry on Duty determination before beginning Government performance
• Be able to obtain the required Government accounts, credentials, badges, and system access
• Comply with SBA and SBA OIG information-security, privacy, evidence-handling, and data-protection requirements
• A bachelor's degree or higher in data engineering, computer science, data science, machine learning, mathematics, or a related field; or at least five years of applied professional experience in one or more of those fields
• At least five years of hands-on experience in maintaining SQL databases
• At least five years of hands-on experience in performing advanced database operations using SQL
• At least five years of hands-on experience in performing advanced database operations using T-SQL
• At least five years of hands-on experience in designing cloud-based ETL and ELT processes
• At least five years of hands-on experience in implementing cloud-based ETL and ELT processes
• At least five years of hands-on experience in maintaining ETL and ELT processes in cloud-based data-analytics environments
• At least three years of hands-on experience in Microsoft Azure Synapse
• At least three years of hands-on experience in Microsoft Azure Machine Learning
• At least three years of hands-on experience in modern cloud data-stack technologies
• At least three years of hands-on experience in data manipulation and processing using Python
• At least three years of hands-on experience in production use of the Pandas library
• Required Technical Skills: Microsoft Azure, Azure Synapse Analytics, Azure Machine Learning, Azure Machine Learning SDK v1 and SDK v2, Azure Data Lake Storage, Python, Pandas, SQL, T-SQL, Microsoft SQL Server, ETL and ELT architecture, Data modeling, Source control, CI/CD, REST APIs, Logging and monitoring, Data validation and quality controls, Parquet and modern data formats, Technical documentation, Data dictionaries and entity-relationship diagrams
• Microsoft DP-203 certification or equivalent Azure data-engineering certification
• Experience with PySpark
• Experience with Polars
• Experience developing reusable and modular Python code
• Experience implementing pipelines and infrastructure through code-first approaches
• Experience using Python SDKs, command-line interfaces, REST APIs, or infrastructure-as-code tools
• Experience implementing source control and CI/CD workflows
• Familiarity with enterprise AI coding assistants
• Familiarity with secure LLM integration patterns
• Experience supporting federal audits, criminal investigations, financial-fraud analytics, or loan-fraud detection
• Experience building and operating production data platforms in regulated or sensitive environments