Geisinger is a healthcare organization seeking a Senior Business Intelligence Analyst to support AI Evaluation & Monitoring. The role involves using analytic expertise to help the organization make data-driven decisions by collecting, cleaning, and interpreting data sets, as well as mentoring junior analysts.
Responsibilities:
- Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes
- Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable — running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work product that governance and program teams can act on
- Collaborates with internal partners and organizations in the design and development of reporting solutions and analytic studies to support business needs
- Plans projects and request completion tasks
- Provides time and effort estimates based on requirements gathering. Strong experience independently managing multiple projects is also required
- Demonstrates working knowledge of relevant source systems needed to support primary stakeholders and proficiency with related databases/data marts within first year in role. Must stay abreast of current industry trends and become familiar with the various business intelligence software systems used at Geisinger
- Recommends analysis approach and additional analyses or reporting needed based on business questions, requirements, and initial analysis findings
- Develops, validates, and executes complex queries. Develops complex reports and performs complex data analysis that answers business questions for internal customers. Interprets results and makes recommendations to business areas based on findings
- Presents analysis and reports to internal customers and stakeholders
- Routinely uses statistical methods during analyses and considers how best to visualize data that accurately reflects the intended message including the limitations
- Reports and helps troubleshoot data quality issues when performing data profiling, testing, or validation
- Collaborative and eager to share knowledge with team members
- Responsible for participating in strategic departmental initiatives
- Acts as a mentor to other analysts
- May participate in industry advisory and/or user groups
- Strong attention to detail and the ability to be precise and clear when describing results of data analyses
- Shares important and relevant information with the team. Proactively offers suggestions, provides resources, volunteers for assignments, and removes barriers to help the team accomplish its goals
- Follows all department's policies and procedures including request intake/fulfillment, change management, and data governance practices
Requirements:
- High School Diploma or Equivalent (GED)- (Required)
- Minimum of 8 years-Relevant experience* (Required)
- Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes
- Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable — running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work product that governance and program teams can act on
- Collaborates with internal partners and organizations in the design and development of reporting solutions and analytic studies to support business needs
- Plans projects and request completion tasks
- Provides time and effort estimates based on requirements gathering. Strong experience independently managing multiple projects is also required
- Demonstrates working knowledge of relevant source systems needed to support primary stakeholders and proficiency with related databases/data marts within first year in role. Must stay abreast of current industry trends and become familiar with the various business intelligence software systems used at Geisinger
- Recommends analysis approach and additional analyses or reporting needed based on business questions, requirements, and initial analysis findings
- Develops, validates, and executes complex queries. Develops complex reports and performs complex data analysis that answers business questions for internal customers. Interprets results and makes recommendations to business areas based on findings
- Presents analysis and reports to internal customers and stakeholders
- Routinely uses statistical methods during analyses and considers how best to visualize data that accurately reflects the intended message including the limitations
- Reports and helps troubleshoot data quality issues when performing data profiling, testing, or validation
- Collaborative and eager to share knowledge with team members
- Responsible for participating in strategic departmental initiatives
- Acts as a mentor to other analysts
- May participate in industry advisory and/or user groups
- Strong attention to detail and the ability to be precise and clear when describing results of data analyses
- Shares important and relevant information with the team. Proactively offers suggestions, provides resources, volunteers for assignments, and removes barriers to help the team accomplish its goals
- Follows all department's policies and procedures including request intake/fulfillment, change management, and data governance practices
- Healthcare, clinical, or regulated-industry data experience strongly preferred
- Applied statistics: hypothesis testing, confidence intervals, power and sample size, regression, and non-parametric methods
- SQL with large, messy clinical data warehouses, including window functions, CTEs, debugging query plans, and performance tuning
- Python for analysis: pandas, numpy, scikit-learn, statsmodels, common evaluation libraries
- Production monitoring design and execution: drift detection, performance decay tracking, adoption and outcome metrics
- LLM and generative AI evaluation execution: golden sets, judge-based scoring, hallucination and grounding checks