Data Engineer

IBM is hiring a Data Engineer to design and deliver hands-on data solutions on AWS and Databricks in collaboration with client teams. The role focuses on data pipelines, platform architecture, legacy workload re-engineering, and migration from MySQL to Databricks Lakebase, while supporting data and AI use cases.

Responsibilities:

• Work onsite or embedded with client teams to understand business problems and turn them into working technical solutions
• Design and build data pipelines and integrations using Fivetran, Databricks, and AWS services
• Lead technical design and delivery for migrating a Tier-1 application from MySQL to Databricks Lakebase, including schema re-design, data migration, cutover, and validation
• Integrate SAP data through SAP BTP and BDC Connect, non-SAP data, and design domain centric data products
• Build prototypes and reference implementations, then work with the client to harden them for production
• Support data and AI use cases, including integration with GCP-based ML/GenAI components
• Establish engineering standards: CI/CD, testing, observability, security, and cost control
• Coach client and delivery team members, and communicate status, risks, and design decisions to stakeholders
• Design end-to-end data solutions across multiple platforms, applying industry standards and best practices to meet client needs
• Lead the development of data platform architectures, including cognitive components and unstructured data technologies, annotation, and related data and analytics
• Manage and maintain large data repositories, ensuring scalability and performance
• Create data solutions on cloud-native platforms (AWS, with GCP for ML/GenAI), tailored to client requirements
• Collaborate with clients to understand their needs and deliver customized data solutions that meet their business objectives

Requirements:

• Bachelor's Degree
• IBM will not be providing visa sponsorship for this position now or in the future. Therefore, in order to be considered for this position, you must have the ability to work without a need for current or future visa sponsorship
• Databricks Certified Data Engineer or Machine Learning Professional
• Advanced hands-on expertise in data engineering and solution architecture, with demonstrated ability to design and deliver complex data solutions
• Strong hands-on expertise with Databricks (Spark, Delta Lake, MLflow, Unity Catalog), AWS, SQL, and Python, including large-scale batch and streaming workloads
• Experience with relational databases such as MySQL, database migration/re-engineering, Fivetran or similar ingestion/ELT technologies, and modern data-sharing patterns
• Experience with Delta Lake, Apache Iceberg, and other open table formats, including schema evolution, optimisation, and interoperability across engines
• Hands-on experience designing scalable data platforms and pipelines using technologies such as Kafka, Amazon EventBridge, or Amazon SQS, along with Terraform and CI/CD practices
• Experience with Databricks Lakebase or similar managed PostgreSQL/OLTP-on-lakehouse technologies
• Knowledge of SAP BTP, BDC Connect, SAP extraction patterns, or SAP IS-U
• Exposure to GCP ML/GenAI services and cloud-native solution design across AWS and GCP
• Experience with Git-based version control and branching strategies for data workloads
• Experience in the utilities or energy industry, with relevant Databricks and/or AWS certification