Sr. Azure Data Engineer///Inperson Interview
Sidley, Austin ,TX
12 Months
Dallas, TX, Onsite (Need to work from 209 State Highway 121 Bypass Suite #36 Lewisville, TX 75067)
Interview process: Internal technical screening , if shortlisted then client interview.
Internal Technical screening is done by third party (CANGRA TALENTS).
interview will be for 45 - 60 minutes and it would be a video interview and will be recorded. Coding will also be part of the interview.
Client Interview : 60 minutes video interview with hiring manager, Coding will also be part of the interview.
Build end-to-end Azure Databricks-based data solutions.
Looking for a strong python experienced engineer with prompt engineering skills.
Client has existing SQL Server based data systems to be converted into data bricks pipelines.
migrate from Snowflake to Data bricks.
client has existing 30 defined pipelines on Snowflake, those needs to be defined/migrated to data bricks.
Looking for candidates who can build end to end Azure data bricks-based data solutions.
Ideal to have data bricks experience or snowflake experience along with Azure.
Role: Senior Data Engineer:
The Senior Data Engineer will design, build, and maintain the scalable data pipelines, models, and infrastructure that power analytics, business intelligence, and machine‑learning products across the company. Partnering closely with business, product, and analytics teams, you will translate complex requirements into elegant, reliable data solutions and help drive the delivery of innovative data products. This role reports to the Senior Manager, Data Engineering.
Duties and Responsibilities:
Build end-to-end Azure Databricks-based data solutions.
Design, develop, and maintain scalable ETL and streaming data pipelines on Azure Databricks, leveraging PySpark, Delta Lake, and Azure Data Lake Storage (ADLS Gen2) to enable reliable Lakehouse architectures and ensure efficient ingestion, transformation, and storage of data
Build and optimize data models and schemas for analytics, reporting, and operational data stores
Build and optimize Delta Lake / Lakehouse patterns (Bronze/Silver/Gold), including schema evolution and time travel
Develop high-quality Python / PySpark / Spark SQL transformations, optimize joins, partitioning, caching, and shuffle behavior.
Implement and maintain data quality frameworks, including data validation, monitoring, and alerting mechanisms.
Collaborate closely with data architects, analysts, data scientists, and product teams to align data engineering activities with business goals.
Leverage cloud data platforms (Azure, AWS, or Google Cloud Platform) to build and optimize data storage solutions, including data warehouses, data lakehouses, and real-time data processing.
Develop automation processes and frameworks for CI/CD supported by version control, linting, automated testing, security scanning, and monitoring
Troubleshoot and resolve complex Azure Databricks platform, data infrastructure and pipeline issues, ensuring minimal downtime and optimal performance.
Experience:
Required:
A minimum of 10 years of hands-on experience in data engineering, designing, and building scalable data pipelines, ETL/ELT processes.
Extensive experience with cloud data platforms in Azure, AWS, or Google
Strong proficiency with Python and SQL for data processing
Proven experience building reusable, metadata-driven data ingestion frameworks using Python and/or Scala
Hands-on experience with modern data-platform components (object storage, Lakehouse engines, orchestration tools, columnar warehouses, streaming services).
Proven experience with data modeling, schema design, and performance tuning of large-scale data systems.
Deep understanding of data engineering best practices: code repositories, CI/CD pipelines, test automation, monitoring, and alerting systems.
Preferred:
Experience building data pipelines in an Azure Databricks environment
Knowledge of Databricks architecture and core components, including Databricks Lakehouse, Delta Lake, Databricks SQL, Apache Spark Clusters, Unity Catalog, Databricks Workflows (Jobs), and Databricks Notebooks.
Proficiency with Apache Spark for data processing
Hands-on experience integrating Azure Databricks with Azure DevOps, Azure Blob Storage / ADLS Gen2, Azure Key Vault, and Azure Data Factory
Experience migrating to or building data platforms from the ground up
Regards,
Satya
Technical Recruiter
Key Business Solutions, Inc||
Experience working in an Agile delivery model
