Sr. Data Engineer

MBO Partners is a deep jobs platform that connects and enables independent professionals and microbusiness owners to do business safely and effectively with enterprise organizations. They are seeking an experienced data engineer to solve applied problems with code and scalable infrastructure, working with large datasets and building robust ETL/ELT pipelines.

Responsibilities:

• You’ll work with large, complex datasets, build robust ETL/ELT pipelines, and optimize data flows using technologies like Databricks, Spark, and Kafka
• With your drive to establish processes and lead technological innovation, you’ll make a lasting impact on the civil market

Requirements:

• 5+ years of experience designing, building, and maintaining data pipelines in production environments
• 3+ years of experience developing with Spark (Databricks preferred), PySpark, or similar distributed systems
• 3+ years of experience with data lakehouse or warehouse platforms, schema design, and query optimization
• 3+ years of experience processing data using streaming (Kafka, Kinesis, etc.) and batch methods
• Proficiency in Python and SQL (Scala or Java a plus)
• Experience implementing best practices for data quality, testing, and observability
• Strong understanding of structured/unstructured data formats (Parquet, Avro, JSON, Delta)
• Knowledge of data, information, and message exchange structures and standards
• Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements
• Bachelor's degree
• Legal authorization to work in the U.S. is required
• 3+ years of experience in data analytics
• Experience designing data flows that leverage a medallion architecture
• Experience with containerization and orchestration (Docker, Kubernetes, EKS)
• Experience with Kafka
• Experience with Nifi
• Experience using AWS
• Knowledge of microservices and integrating with data services
• Knowledge of database schema design
• Master's degree