Judge Won’t Overturn Invalidated USMLE Scores

Calling the matter “a danger the public should not be forced to shoulder,” a federal judge has rejected a plea to temporarily restore the scores of 832 medical graduates from Nepal who are suspected of cheating on the United States Medical Licensing Exam (USMLE).

In a February 23 order, Judge Christopher R. Cooper, of the US District Court for the District of Columbia, denied Latika Giri’s emergency motion to block the National Board of Medical Examiners (NBME) from invalidating the scores, ruling the public interest plainly weighs against granting the request.

“First and foremost, is the overriding interest in public safety,” Cooper wrote in his 32-page order. “This is a case about the credentials of doctors applying to medical residency programs…Granting the preliminary injunction would create an unacceptable risk that individuals who lack the requisite knowledge and skills they purport to possess because they achieved their exam scores fraudulently will be administering medical care to unsuspecting patients across the nation.”

Attorneys for Giri did not return messages seeking comment about the order.

The NBME also did not return messages seeking comment. The board previously said it does not comment on pending litigation.

The decision is the latest development in a widespread cheating scandal first reported by Medscape Medical News. Giri, an international medical graduate (IMG) from Kathmandu, sued NBME earlier this month claiming the board discriminated against Nepali medical graduates when it invalidated hundreds of exam scores linked to the country.

Giri also accused NBME of violating its own procedures when it voided the scores before giving examinees a chance to argue and appeal. She asked the district court to block NBME from invalidating her exam scores while the lawsuit continues and restore her original results.

In court documents, NBME argued that it did not invalidate the scores because the examinees were Nepali but because staff concluded that there was “a good faith basis for questioning the validity of the scores.”

The invalidations were based on concerns that the results reflected prior access to secure exam content rather than knowledge and understanding of the medical principles and skills the exams are intended to assess, according to the NBME’s legal response.

“The USMLE program took reasonable and appropriate actions to prevent the significant harm and disruption that would result from allowing potentially unqualified individuals to participate in the 2024 residency Match,” the NBME stated in court documents. “If granted, the requested injunction would cause enormous harm not only to NBME…but also to state licensing authorities, which rely upon USMLE results to help ensure that physicians have the minimum competencies needed to provide safe and effective health care.”

In his order, Cooper wrote that Giri has not proven the board’s actions were discriminatory against Nepali doctors.

“Nothing in the present record suggests that NBME went looking for a problem in Nepal out of ethnicity-or national-origin based [sic] suspicion,” Cooper wrote. “[It] followed the trail of evidence, including tips about organized cheating taking place in medical schools and at a testing center located in Nepal, and on an online forum for which a ‘nexus to Nepal’ was a ticket to admission.”

NBME: Nepal Outperformed All Other Countries on USMLE

Court documents shed more light on NBME’s investigation into the suspected cheating and on the anomalous patterns the board allegedly discovered from Nepal medical graduates.

In response to anonymous tips, the USMLE program in early 2023 asked the NBME Psychometrics and Data Analysis (PADA) unit to analyze examinee performance data for test centers in Jordan, Nepal, and Pakistan, according to court records. Within the initial data analysis, the data involving the single test center in Nepal was “the most extreme,” the unit found.

Out of more than 400 test centers across the world, including those in the United States, the test center in Nepal produced the highest test scores in the world for Step 1 in 2021 and 2022 and the highest test scores in the world for Step 2 CK in 2022, according to court documents. For the 2022 Step 1 exam for example, the average score of examinees testing in the Nepal test center was 240. No other test center in the world had an average examinee score above 227, according to the NBME’s legal response.

The median item response time for examinees who tested at the Nepal test center in 2022 was also among the fastest of all international test centers for Step 1 and Step 2 CK, investigators found.

In addition, the volume of examinees taking the USMLE Step 1 and Step 2 CK at the Nepal test center in Nepal had sharply increased. Step 1 volume more than doubled in the Nepal test center from 281 examinees in 2019 to 662 examinees in 2022, according to court documents.

The rapid increase continued in 2023, when examinee volume was nearly three-and-a-half times higher than the 2019 volume. The data were consistent with anonymous tips received by the USMLE program office, suggesting there may be wide-scale collection and sharing of live USMLE exam content within Nepal.

Investigation Finds Similar Correct and Incorrect Answers

Agreement similarity among the exams analyzed also raised red flags. Investigators ran an “agreement analysis” for all examinees who tested at centers in Jordan, Nepal, and Pakistan as well as two centers in India, according to court documents.

For the 2022 Step 1 exam and the 2021 and 2022 Step 2 CK exam, the analysis showed a substantially higher percentage of examinees with a statistically significant level of agreement matches in the examine group that tested at centers in Jordan, Nepal, Pakistan, and India compared with the baseline group, according to legal records.

The vast majority of examinees with a statistically significant number of matching incorrect answers tested at the Nepal test center, data showed.

Further analysis found that examinee volumes increased considerably at the Nepal test center in the months prior to the USMLE program releasing new test items, “suggesting that candidates who had prior access to disclosed exam questions wanted to test before new questions came into the item pool.”

Investigators also identified posts on social medial and in online chat rooms suggesting groups were collecting and sharing large amounts of secure exam material in private groups. Some posts advised examinees to use the full examination time when taking the USMLE “to avoid raising suspicion about having had prior access to secure exam materials,” according to court documents.

From its investigation and analysis, the USMLE program identified 832 examinees who had passing level exam results whose validity the USMLE program had a significant and good faith basis for questioning, according to court records.

Of the total, 618 examinees had one Step score flagged as being of questioned validity, 202 examinees had two Step exam scores flagged, and 12 examinees had scores flagged on all three Step exams.

NBME Defends Departure From Traditional Procedures

In court documents, NBME disputed claims that it violated its own procedures by invalidating the exam scores. Giri’s report contends that examinees suspected of cheating are typically first advised of the matter, given an opportunity to share relevant information, and provided the right to appeal — during which time, their scores are treated as valid.

But the NBME said the USMLE program is authorized to take any actions it deems appropriate in response to concerns regarding score validity if the USMLE Committee for Individualized Review or the USMLE Composite Committee concludes that alternative or supplemental procedures are warranted in response to a given set of facts or circumstances.

“Following the month-long investigation and analysis…the USMLE program concluded that alternative procedures were warranted to address the score invalidity concerns identified in the interest of providing a process that is timely, efficient, effective, and fair, and given the large number of examinees involved in the investigation,” the board stated in its legal response.

In his order, Cooper wrote the current scenario, which implicates that more than 800 test-takers, is “clearly a situation calling for a procedure geared toward efficiency.” No evidence shows the board would not have taken similarly swift action if confronted with evidence of cheating on a comparable scale elsewhere, he wrote.

The judge also denied Giri’s motion to certify the lawsuit as a class action. The motion was denied without prejudice, meaning the plaintiff has the option to renew the motion should the case proceed.

Alicia Gallegos is a freelance healthcare reporter based in the Midwest. She can be reached on X, formally known as Twitter: @legal_med