Senior Data Engineer - SDE 26-04140

Job Title: Senior Data Engineer

Location: New York, NY 10020

Position Summary

We are seeking a highly experienced Senior Data Engineer to support the design, development, and optimization of large-scale data platforms within a financial services environment. This role focuses on building robust data pipelines, enabling advanced credit risk analytics, and ensuring compliance with regulatory standards.

The ideal candidate will bring deep expertise in credit and counterparty risk, modern data engineering technologies, and cloud-based data platforms, while collaborating closely with cross-functional teams including risk, quant, and compliance stakeholders.

Key Responsibilities

<>Data Architecture & Engineering</>

• Lead architecture and technical design discussions for credit risk data platforms
• Design and implement scalable batch and streaming data pipelines using PySpark
• Build ingestion frameworks for structured and unstructured data from upstream systems into cloud storage (e.g., S3/ADLS)
• Implement data processing workflows using Medallion Architecture (Bronze, Silver, Gold layers)

<>Data Processing & Optimization</>

• Develop and optimize transformation logic for large-scale financial datasets
• Ensure high data quality, auditability, and regulatory compliance
• Apply optimization techniques such as partitioning, indexing, and schema evolution
• Model and optimize risk metrics (e.g., PD, LGD, EPE, PFE, CVA) for analytics and reporting

<>Platform Integration & Operations</>

• Integrate with external risk engines and support orchestration of complex batch processes
• Ensure platform reliability, observability, and data lineage tracking
• Implement and maintain security standards (IAM, encryption, authentication protocols)
• Troubleshoot production issues and provide ongoing support

<>Collaboration & Documentation</>

• Collaborate with data scientists, risk analysts, and business stakeholders
• Contribute to API design and data contracts for internal and external consumers
• Maintain comprehensive technical documentation for audit and compliance purposes
• Participate in Agile development processes and ceremonies

Required Qualifications

• 12+ years of experience in data engineering or data development
• Strong experience within financial services, particularly credit or counterparty risk
• Expertise in regulatory frameworks such as Basel III/IV, IFRS 9, CECL, or FRTB
• Advanced proficiency in Python and PySpark / Apache Spark
• Hands-on experience with cloud platforms and data lake technologies (e.g., Databricks, Delta Lake)
• Strong SQL skills including complex queries, joins, and performance optimization
• Experience building data pipelines from multiple financial data sources
• Familiarity with workflow orchestration tools (e.g., Airflow or similar)
• Experience with CI/CD tools such as Git, Jenkins, or Azure DevOps
• Cloud certification (e.g., AWS Certified Cloud Practitioner or equivalent)

Preferred Qualifications

• Experience with large-scale financial risk systems and analytics platforms
• Background in designing data architectures and maintaining data dictionaries
• Experience working in Agile environments using tools such as JIRA or Confluence
• Exposure to modern data governance and compliance practices

Skills & Competencies

• Strong analytical and problem-solving abilities
• Excellent communication skills with both technical and non-technical stakeholders
• Ability to work collaboratively with cross-functional teams
• High attention to detail and commitment to data quality
• Proactive, self-driven mindset with continuous learning approach

Work Environment

• Fast-paced, data-driven financial services environment
• High level of collaboration with global teams
• Focus on regulatory compliance, data governance, and innovation