Business Intelligence Analyst Senior (AI Evaluation & Monitoring)

Geisinger is a healthcare organization seeking a Senior Business Intelligence Analyst to support AI Evaluation & Monitoring. The role involves using analytic expertise to help the organization make data-driven decisions by collecting, cleaning, and interpreting data sets, as well as mentoring junior analysts.

Responsibilities:

• Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes
• Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable — running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work product that governance and program teams can act on
• Collaborates with internal partners and organizations in the design and development of reporting solutions and analytic studies to support business needs
• Plans projects and request completion tasks
• Provides time and effort estimates based on requirements gathering. Strong experience independently managing multiple projects is also required
• Demonstrates working knowledge of relevant source systems needed to support primary stakeholders and proficiency with related databases/data marts within first year in role. Must stay abreast of current industry trends and become familiar with the various business intelligence software systems used at Geisinger
• Recommends analysis approach and additional analyses or reporting needed based on business questions, requirements, and initial analysis findings
• Develops, validates, and executes complex queries. Develops complex reports and performs complex data analysis that answers business questions for internal customers. Interprets results and makes recommendations to business areas based on findings
• Presents analysis and reports to internal customers and stakeholders
• Routinely uses statistical methods during analyses and considers how best to visualize data that accurately reflects the intended message including the limitations
• Reports and helps troubleshoot data quality issues when performing data profiling, testing, or validation
• Collaborative and eager to share knowledge with team members
• Responsible for participating in strategic departmental initiatives
• Acts as a mentor to other analysts
• May participate in industry advisory and/or user groups
• Strong attention to detail and the ability to be precise and clear when describing results of data analyses
• Shares important and relevant information with the team. Proactively offers suggestions, provides resources, volunteers for assignments, and removes barriers to help the team accomplish its goals
• Follows all department's policies and procedures including request intake/fulfillment, change management, and data governance practices

Requirements:

• High School Diploma or Equivalent (GED)- (Required)
• Minimum of 8 years-Relevant experience* (Required)
• Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes
• Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable — running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work product that governance and program teams can act on
• Collaborates with internal partners and organizations in the design and development of reporting solutions and analytic studies to support business needs
• Plans projects and request completion tasks
• Provides time and effort estimates based on requirements gathering. Strong experience independently managing multiple projects is also required
• Demonstrates working knowledge of relevant source systems needed to support primary stakeholders and proficiency with related databases/data marts within first year in role. Must stay abreast of current industry trends and become familiar with the various business intelligence software systems used at Geisinger
• Recommends analysis approach and additional analyses or reporting needed based on business questions, requirements, and initial analysis findings
• Develops, validates, and executes complex queries. Develops complex reports and performs complex data analysis that answers business questions for internal customers. Interprets results and makes recommendations to business areas based on findings
• Presents analysis and reports to internal customers and stakeholders
• Routinely uses statistical methods during analyses and considers how best to visualize data that accurately reflects the intended message including the limitations
• Reports and helps troubleshoot data quality issues when performing data profiling, testing, or validation
• Collaborative and eager to share knowledge with team members
• Responsible for participating in strategic departmental initiatives
• Acts as a mentor to other analysts
• May participate in industry advisory and/or user groups
• Strong attention to detail and the ability to be precise and clear when describing results of data analyses
• Shares important and relevant information with the team. Proactively offers suggestions, provides resources, volunteers for assignments, and removes barriers to help the team accomplish its goals
• Follows all department's policies and procedures including request intake/fulfillment, change management, and data governance practices
• Healthcare, clinical, or regulated-industry data experience strongly preferred
• Applied statistics: hypothesis testing, confidence intervals, power and sample size, regression, and non-parametric methods
• SQL with large, messy clinical data warehouses, including window functions, CTEs, debugging query plans, and performance tuning
• Python for analysis: pandas, numpy, scikit-learn, statsmodels, common evaluation libraries
• Production monitoring design and execution: drift detection, performance decay tracking, adoption and outcome metrics
• LLM and generative AI evaluation execution: golden sets, judge-based scoring, hallucination and grounding checks