EPAM SystemsRemote
Senior Data Integration Engineer
United States of America
Full Time
3 weeks ago
H1B Sponsor Likely
Key skills
PythonSQLETLELTData EngineeringData WarehousingSnowflakeRedshiftSparkApache SparkKafkaAirflowApache AirflowdbtAWSAzureGCPKubernetesDockerGitHub ActionsGitLab CIServerlessLambdaGlueAzure FunctionsAzure DevOpsPostgreSQLSQL ServerRESTfulGitGitHubGitLabVersion ControlRepositoryCI/CDCollaboration
About this role
EPAM Systems is seeking a highly skilled, solution-oriented Senior Data Integration Engineer with deep expertise in modern data engineering and cloud-native architectures. In this role, you will be responsible for creating, optimizing, and modernizing advanced data integration systems while collaborating with product, engineering, and architecture teams.
Responsibilities:
- Pipeline Architecture & Development: Design, build, and optimize scalable, reliable batch and near-real-time ETL/ELT pipelines using Python, PySpark, SQL, and modern cloud integration engines
- Orchestration & Automation: Develop and manage complex workflow orchestrations (using Apache Airflow or cloud native schedulers) and automate ingestion routines to minimize manual operations
- Data Modeling & Warehousing: Design and implement modern data warehouse/lakehouse layers (using Snowflake, ClickHouse, Azure Synapse, or Redshift), establishing optimal partitioning, indexing, and Slowly Changing Dimension (SCD Type 2) patterns
- Data Quality & Testing Integration: Establish rigorous data quality checks and validation frameworks utilizing tools like dbt (data build tool), Soda, or customized PySpark testing suites
- Collaboration & Design: Work closely with product owners, business analysts, and systems architects to define data requirements, analyze technical constraints, design Source-to-Target Mappings (STTM), and make critical architectural decisions
- Code Quality & DevOps: Maintain a clean, modular code repository. Lead code reviews, enforce engineering standards, and configure robust CI/CD pipelines (Azure DevOps, GitLab CI, or GitHub Actions) with Docker containers
- Technical Documentation: Deliver comprehensive, clear technical specs, metadata lineage documentation, architectural diagrams, and data dictionaries
Requirements:
- 5+ years of hands-on experience in data engineering, data warehousing, database design, and end-to-end data integration
- Advanced knowledge of Cloud Integration tools such as Azure Data Factory (ADF), AWS Glue, or GCP Dataflow
- Proficiency in workflow orchestrators like Apache Airflow and exposure to CDC (Change Data Capture) or real-time streaming tools (e.g., Kafka, Debezium)
- Strong production-level coding skills in SQL (advanced optimization/stored procedures), Python, and PySpark / Apache Spark
- Experience working with high-performance databases and cloud-native systems (e.g., Snowflake, ClickHouse, PostgreSQL, MS SQL Server, or Azure Synapse)
- Master-level understanding of data modeling practices (OLAP, OLTP, Star/Snowflake schemas, Delta Lake/Lakehouse patterns, and Data staging processes)
- Hands-on experience with version control (Git) and building automated deployment pipelines (CI/CD) for data products
- Proven ability to articulate complex technical ideas clearly to both business stakeholders and developers. Fluency in English (Upper-Intermediate level or higher)
- Deep knowledge of dbt (data build tool) and schema validation practices
- Experience using Docker or Kubernetes to package and deploy data applications
- Experience building lightweight, serverless ingestion services (e.g., using AWS Lambda / Azure Functions and RESTful APIs)