Data Validation Process

©️ 2024 Kaiser Foundation Health Plan, Inc.

This resource is part of the Data Quality & Reporting Implementation Guide, offering steps and activities to ensure your practice is capable of reporting valid and reliable data for selected population health measures. It is the first in the “Building the Foundation” series of implementation guides.

Validating your data is important to ensure that each rate is an accurate, reliable reflection of the care that has been delivered and the outcomes a patient has experienced. Through the Population Health Management Initiative (PHMI), community health centers (CHCs) should have a validation process in place to ensure the accuracy of PHMI/HEDIS measurement.

This document delineates an approach for basic data validation and key steps to validate performance metric rates for the seven core HEDIS measures for PHMI. This process includes general steps (e.g., validating eligible population and stratified sub-populations), as well as steps specific to each core measure. Increased confidence in the accuracy of the rates reported will lead to better decisions regarding performance improvement interventions and ongoing monitoring.

For purposes of this document, data validation is the act of reviewing and confirming that the CHC data used to calculate the measures are of a minimum acceptable level of quality and accuracy. Depending on the analytics platform and process used to calculate each measure, validation can be an automated or a manual process or a combination.

Data validation can occur at each step in the calculation process of a measure to ensure the accuracy of that specific component.

Validation includes:

• Overall checks to verify the reasonability of the final rate (such as reasonability of the numerator and denominator).
• Specific checks to confirm that a service or visit was correctly coded on a member level.

The process presented here could also be applied to other quality measures, including PHMI supplemental measures, other non-PHMI measures or as part of a broader data governance program.

Reliable measurement is critical to the success of the PHMI implementation. Prior to reporting the core HEDIS measures for PHMI, the CHC should validate the numerator, denominator and overall eligible population for each measure.

Frequency of Validation and Confirmation

The level of rigor needed for ongoing data validation depends on successful validation and remediation of gaps in prior reporting periods and the level of automation or built-in validation in the analysis software/platform.

In general, CHCs who are using less automation should plan for a full validation of data with every submission. These CHCs could, over time, increase their levels of automated data and reduce the amount of manual validation. A CHC that uses an application to extract measurement data and has software with built-in certified data measures could expect to perform less validation with each submission.

Factors that will be considered in determining the frequency and level of data validation needed could include whether:

• There are no known gaps in reliability/accuracy of data from prior submissions/ validations.
• The CHC is using an analytics software/platform with certified HEDIS measure source codes (see certified vendor list).^[1]
• The CHC is using an analytics software/platform with robust built-in validation checks.
• The CHC has coded processes for extraction of measurement data.
• The CHC has robust data governance processes and procedures and compliance with these processes.

CHCs not meeting the above criteria should conduct a rigorous data validation process of the core HEDIS measures for PHMI. With each submission of PHMI rates, CHCs will be asked to attest that they have validated their measure rates.

Passing Data Validation

A robust data validation process will allow CHCs to identify and remediate issues with data quality and reliability and ensure measure rates are an accurate reflection of patient care received and services provided. A minimum threshold for “passing” data validation standards is an impact of less than 5% percent on the rate. Impacts can be estimated based on the results of the data checks and primary source verification described below.

If data validation results indicate a known deviance greater than 5%, CHCs should:

1. Indicate data validation issues on their Data Reporting Tool (DRT) submission.
2. Develop and implement a remediation plan.

As a best practice, CHCs should migrate from a manual validation process to a mostly automated process. Adopting a more automated approach offers a faster, more efficient and consistent way to extract the data for the core HEDIS measures for PHMI. It also offers improved data quality and integrity in other business processes and programs, like CalAIM or MCP P4P.

Even with robust software, some of the steps below would still apply to ensure the quality and accuracy of the CHC’s core HEDIS measures for PHMI. Understanding where there are strong validation steps in place and where there are gaps will be critical to determining a process going forward.

In conjunction with their practice coach and the data quality and reporting subject matter experts (SMEs), the following is a general process that CHCs could follow to assess and improve their validation process:

Step 1: Analyze current state and identify software/methodology for producing each measure.

• Which software, if any, is the CHC currently using for each measure?
• Which version (i.e., are they using the current version)?
• Are built-in validation checks or processes available? If so, for what?

Step 2: Analyze overall data governance.

• What is the CHC’s overall data governance structure?
• What policies and procedures are in place to ensure data governance?
• How is the integrity and validation of data ensured by the CHC’s data governance structure?

Step 3: Determine gaps and level of data validation needed.

• Are there gaps in automated processes not covered by manual processes?
• Are there manual procedures that are not documented or consistently followed?
• Does the CHC lack a procedure to perform a data validation function?

Step 4: Explore gap remediation approaches.

• If not using one currently, is there an interest/ability in using an analytics option with certified measure reporting?
• A resource: directory of vendors that have earned measure certification.
• Identify equivalent manual processes to complete functions/fill gaps, including introducing policies and procedures as necessary.

Step 5: Develop sustainable processes for ongoing data integrity and validation.

• Ensure sustainable processes and develop policies and procedures as needed.

The clinic population is the basis from which all measures are derived. PHMI/HEDIS measures are reported for all Medi-Cal MCP assigned patients to the clinic. Ensuring that this population is accurate in the database or master file from which the measures are calculated is critical to ensuring the accuracy of the eligible populations for each measure and the segmented populations within each.

Each measure then requires validation of its unique data components, including numerators and denominators, to ensure the reliability and accuracy of the measure. Validation should consist of:

1. Hard checks for data discrepancies.
2. Reasonability checks for logical consistency.
3. Primary source verification.

Hard Checks

A hard check identifies discrepancies that are mathematically incongruent (e.g., a single race/ethnicity should not have a larger population than the total population).

Reasonability Checks

Reasonability checks determine whether the data is logical or consistent with other knowledge (e.g., a rate that is logical based on what is known about the patient population from other data metrics or sources).

When identifying the reasonability of populations, numerators and denominators, CHCs should consider what they know, such as:

• What is their knowledge of their patient population and do the numbers align?
• What other similar metrics does the CHC report (e.g., UDS) and do the core set measures (which utilize HEDIS specifications) seem reasonable in comparison?
• How do the CHC rates compare to regional state, or national benchmarks?
• How do the CHC rates compare to prior reporting periods?
• How do they compare to rates calculated by MCPs?

These questions will help to determine the “appearance” of reasonability. Some variation may be expected (for instance, UDS and HEDIS specifications differ), however, this process could detect differences the CHC would not expect to see based on the different specifications alone.

Detecting Expected Versus Unexpected Differences

Example 1: A CHC may know through other measures or trends that approximately 25% of the patient population has diabetes. If this prevalence is known, a CHC could determine that a denominator for the diabetes measure showing 5% or 95% of the health clinic population has diabetes does not appear reasonable.

Example 2: Compared to a prior reporting period of the same measure with no change in the specifications, neither the numerator nor the denominator should change significantly without another known change.

• A numerator can increase if process improvement activities are underway to improve the measure.
• A denominator can increase if there is a significant change in a CHC’s population.

Numbers should not vary significantly without an explanatory factor.

Use the below table as a checklist to validate the overall population in the database or master file from which the measures are calculated, as well as the rates for each core HEDIS measure for PHMI.

FIGURE 7.1: VALIDATION CHECKLIST 1: HARD CHECKS AND REASONABILITY CHECKS

Primary Source Verification

Primary source verification (PSV) is a common best practice. For example, health plans reporting HEDIS measures must go through PSV with a HEDIS auditor for certain types of data to ensure its accuracy and reliability for use in reporting.

PSV involves:

• Using the patient-level data file for a measure. This is a file that identifies all patients in the denominator with a flag indicating whether they were numeratorcompliant.
• Tracing that patient back to their primary source of data (e.g., the medical record) to ensure that the documentation supports the patient having been included in the measure and whether they are compliant.

An Example of Primary Source Verification

A patient who was identified as numerator-compliant for the colorectal cancer screening measure should have evidence in the medical record of factors that comply with the measure. This would include:

• Patient is 45 to 75 years of age.
• Patient has documented evidence of a colorectal cancer screening within the time frame allowed for the particular type of colon cancer screening received.

Recommended Approach to Primary Source Verification

PSV can be a resource-intensive process. For PHMI, initial PSV could consider a random sample of five patients from each measure. If the primary source verified the information contained in the patient-level file for those patients, the measure would pass PSV. If discrepancies were detected, the CHC should:

• Select a larger sample (up to an additional 45 records, as a best practice).
• Assess results of the larger sample to determine if deficiencies are detected.
  • If no, PSV is complete.
  • If yes, the CHC should identify gaps and determine whether the gaps are isolated or pervasive.

Based on findings, the CHC should develop an appropriate remediation strategy and report data validation issues on its data reporting tool (DRT) submission. CHCs should use PSV to validate their PHMI/HEDIS rates:

• As an initial process and continuing with each submission until no issues are found during PSV.
• Any time there is a material change in how the measures are pulled or data sources used.

Use Figure 7.2: Validation Checklist 2: Primary Source Verification below as a checklist when conducting primary source verification of the overall population in the database or master file from which the measures are calculated, as well as rates for each core HEDIS measure for PHMI.

FIGURE 7.2: VALIDATION CHECKLIST 2: PRIMARY SOURCE VERIFICATION