Showing posts with label signaling. Show all posts
Showing posts with label signaling. Show all posts

Sunday, December 10, 2023

Signaling for residency programs in dermatology, general surgery, and internal medicine

We're starting to see some data from signaling for residency applications.  This paper observes that programs are more likely to interview candidates who send them a signal. (Economists will worry that this reflects which programs are signaled and not just the effect of a signal...)  These three specialties have relatively few signals, more like economics than like Orthopedic Surgery (which has 30 signals).  And the table indicates that more interviews are offered than signals received, so that's another difference from Ortho...)

Rosenblatt, Adena E., Jennifer LaFemina, Lonika Sood, Jennifer Choi, Jennifer Serfin, Bobby Naemi, and Dana Dunleavy. "Impact of Preference Signals on Interview Selection Across Multiple Residency Specialties and Programs." Journal of Graduate Medical Education 15, no. 6 (2023): 702.

"Abstract

"Background Program signaling is an innovation that allows applicants to express interest in specific programs while providing programs the opportunity to review genuinely interested applicants during the interview selection process.

"Objective To examine the influence of program signaling on “selected to interview” status across specialties in the 2022 Electronic Residency Application Service (ERAS) application cycle.

"Methods Dermatology, general surgery-categorical (GS), and internal medicine-categorical (IM-C) programs that participated in the signaling section of the 2022 supplemental ERAS application (SuppApp) were included. Applicant signal data was collected from SuppApp, applicant self-reported characteristics collected from the MyERAS Application for Residency Applicants, and 2020 program characteristics collected from the 2020 GME Track Survey. Applicant probability of being selected for interview was analyzed using logistic regression, determined by the selected to interview status in the ERAS Program Director’s WorkStation.

"Results Dermatology had a 62% participation rate (73 of 117 programs), GS a 75% participation rate (174 of 232 programs), and IM-C an 86% participation rate (309 of 361 programs). In all 3 specialties examined, on average, signaling increased the likelihood of being selected to interview compared to applicants who did not signal. This finding held across gender and underrepresented in medicine (UIM) groups in all 3 specialties, across applicant types (MDs, DOs, international medical graduates) for GS and IM-C, and after controlling for United States Medical Licensing Examination Step 1 scores.

"Conclusions Although there was variability by program, signaling increased likelihood of being selected for interview without negatively affecting any specific gender or UIM group."



Data from future years will be needed to determine how signaling is influencing the distribution of residents to programs.

Wednesday, December 6, 2023

Applying for medical residencies: a consensus statement from Internal Medicine

 The Alliance for Academic Internal Medicine has released a "consensus statement" with many proposals about application and interview caps, and signaling.

Catalanotti, Jillian S., Reeni Abraham, John H. Choe, Kelli A. Corning, Laurel Fick, Kathleen M. Finn, Stacy Higgins et al. "Rethinking the Internal Medicine Residency Application Process to Prioritize the Public Good: A Consensus Statement of the Alliance for Academic Internal Medicine." The American Journal of Medicine (2023).

It also includes a call for data and analysis:

"AAIM proposes increasing internal medicine program preference signals to 15, using tiered signaling with three “gold” and 12 “silver” signals, and setting an interview cap of 15 in the 2024-2025 recruitment season, with participation by all internal medicine programs. The Alliance recommends that all internal medicine programs participate in ACI. AAIM recommends that programs transparently share information about their use of preference signals and other application screening methods and calls for real-time data analysis to explore impact, inform future iterations and identify potential harms.

"The Alliance calls upon ERAS and NRMP as well as Thalamus® and other interview scheduling platforms to transparently share data, to embrace change, and to perform analyses needed to inform this process. For example, recent modeling with eight years of retrospective NRMP data in OBGYN demonstrated that an early match round may increase the number of “mutually dissatisfied applicant-program pairs” and that a multiple-round match process could introduce potential rewards for gamesmanship, a prime factor addressed by the current process.35 AAIM applauds this analysis and hopes that the new collaboration between ERAS and Thalamus® may provide useful interview data to inform this proposal and further interventions."

And here is reference 35 in that last paragraph, about which I've blogged before.

I Ashlagi, E Love, JI Reminick, AE. Roth
Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021
J Grad Med Educ, 15 (2) (Apr 2023), pp. 219-227, 10.4300/JGME-D-22-00177.1

Friday, September 8, 2023

Signaling for Orthopaedic surgery residencies

 The Journal of Bone and Joint Surgery reports a small survey about signaling behavior among applicants for orthopaedic surgery residencies.

Deckey, David G., Eugenia Lin, Coltin RB Gerhart, Joseph C. Brinkman, Karan A. Patel, and Joshua S. Bingham. "Decoding the Signals: An Analysis of Preference Signaling in the 2023 Orthopaedic Surgery Residency Match." JBJS Open Access 8, no. 3 (2023).

"While previously used in other specialties, the preference signaling program (PSP) was implemented in the 2022 to 2023 orthopaedic surgery residency application process for the first time. The PSP allowed for 30 signaling tokens to be sent by applicants to programs of their choice to indicate particular interest in a program.

...

"An anonymous electronic survey was emailed to all orthopaedic surgery residency applicants who applied to the authors' institution during the 2022 to 2023 application cycle. The survey was sent after match lists were submitted and closed before the release of match results. 

...

"The survey was completed by 101 applicants. Applicants applied to a mean of 90 programs (range: 10-197) and received an average of 12 interview invitations (range: 0-39). Applicants almost uniformly used all 30 signals, with nearly two-thirds signaling their home programs (65%, 49/76), and nearly all applicants sending signals to programs at which they performed away rotations (95.7%, 88/92). Applicants received a mean of 9 invitations from programs they signaled, compared with 2 invitations from programs they did not signal."

#######

Applicants report sending signals to all the programs that would have been expected to automatically give them interviews even in the absence of a signaling mechanism--namely their home programs and those which they have spent time visiting in 'away rotations.'

In the Economics job market, which may have been the first to introduce signaling, we limited applicants to 2 signals, and advised them not to signal jobs in which they already had well established mutual interests, i.e. not to signal jobs which they felt would interview them without signals. One of the ideas behind the Econ signaling mechanism is that there are many ways applicants can send signals of interest within a network to which they are well connected. Since some applicants are better connected than others, we were offering some signals that could be sent out of network.

It will be interesting to understand if signals to Ortho residencies are helping promote out of network interviews, or are largely strengthening the network connections already established by medical schools (home programs) and away rotations.

Tuesday, August 29, 2023

OB-GYN doctors will use a new application system to apply to residency programs (but will continue to go through the NRMP resident match)

 Before new doctors can participate in the resident match (by engaging with the NRMP), they first have to apply to residency programs, and arrange interviews.  This process has been experiencing congestion, and the specialty of Obstetrics and Gynecology has now decided to switch application services. 

However, participation in the NRMP will not change: the Association of Professors of Gynecology and Obstetrics (APGO) FAQ states "Obstetrics and gynecology applicants will use the National Residency Match Program (NRMP) for the Match. This new application does not change how the applicant or programs interact with the NRMP Match system."

Medpage Today has the story:

Ob/Gyn Switching to Independent System for Residency Applications— This is the last year ob/gyn will use ERAS, despite helping to pilot the program  by Rachael Robertson, Enterprise & Investigative Writer, MedPage Today August 25, 2023

"Beginning next year, ob/gyn programs will start using an independent system for processing residency applications, rather than the Electronic Residency Application Service (ERAS).

"The joint decision to switch to the new system was made by the American College of Obstetricians and Gynecologists (ACOG), the Association of Professors of Gynecology and Obstetrics (APGO), and the Council on Resident Education in Obstetrics and Gynecology (CREOG). The new system will be managed by Liaison International, which uses "Centralized Application Service (CAS) technology," according to the company's website.

...

"A joint statement on the APGO website opens in a new tab or window

said that the new system "will be user friendly and efficient, less expensive for applicants, and will directly decrease the burdens faced by program directors, program managers, and applicants alike," and "will incorporate the entirety of interview season functions, from application submission, review, interview offers and interviews, to rank list submission."

ACOG explained that the decision to pull the ERAS stemmed from the Right Resident, Right Program, Ready Day One initiativeopens in a new tab or window, noting that the new system is mobile-friendly and "will include immediate fee reduction," as detailed on their FAQ pageopens in a new tab or window.

In response, the Association of American Medical Colleges (AAMC), which runs ERAS, issued a statementopens in a new tab or window attributed to President and CEO David J. Skorton, MD, and Alison J. Whelan, MD, the chief academic officer, saying they were "surprised and dismayed" by the decision. 

...

"Bryan Carmody, MD, of Eastern Virginia Medical School in Norfolk, shared information  opens in a new tab or window

about the change on social media, writing on his blogopens in a new tab or window that ob/gyn program directors helped to pilot ERAS when it was first rolled out in the mid-90s.

Carmody told MedPage Today that he anticipates the biggest downsides will fall on applicants, such as those who want to apply to another specialty in addition to ob/gyn.

"Those applicants will have to use one system to apply to ob/gyn and another to their other specialty," he explained. "The same thing applies to applicants who fail to match. They'll have to use ERAS to apply to another specialty during SOAP [Supplemental Offer and Acceptance Program] since few, if any, ob/gyn positions are typically available."

*************

OB-GYN will continue to employ signaling  (very loosely modeled on the signaling used in the Econ PhD job market, but asking applicants to submit 3 "gold" signals and 15 "silver" signals): 

Program Signaling for OBGYN Residency Application Background and FAQs

**************

Earlier:

Friday, April 21, 2023

Friday, April 21, 2023

Transition from medical school to residency: defending the parts that work well (namely the NRMP Resident Match)

This post is about a recently published paper concerning the design of the market for new doctors in the U.S.  But it will require some background for most readers of this blog.   The short summary is that the market is experiencing problems related to congestion, and one of the proposals to address these problems was deeply flawed, and would have reduced market thickness and caused substantial direct harm to participants if implemented, and created instabilities that would likely have caused indirect harms to the match process in subsequent years. But this needed to be explained in the medical community, since that proposal was being  very actively advocated.

For those of you already steeped in the background, you can go straight to the paper, here.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

If the title doesn't remind you of the vigorous advocacy for an early match for select positions, here is some of the relevant back story.

The market for new doctors--i.e. the transition from medical school to residency--is experiencing growing pains as the number of applications and interviews has grown, which imposes costs on both applicants and residency programs.  

Below is a schematic of that process, which begins with applicants submitting applications electronically, which makes it easy to submit many.  This is followed by residency programs inviting some of their applicants to interview. The movement to Zoom interviews has made it easier to have many interviews also (although interviews were multiplying even before they moved to Zoom).  

After interviews, programs and applicants participate in the famous centralized clearinghouse called The Match, run by the NRMP. Programs and applicants each submit rank order lists (ROLs) ranking those with whom they interviewed, and a deferred acceptance algorithm (the Roth-Peranson algorithm) produces a stable matching, which is publicly announced on Match Day. (Unmatched people and positions are invited into a now computer-mediated scramble, called SOAP, and these matches too are announced on Match  Day.)

The Match had its origins as a way to control the "unraveling" of the market into inefficient bilateral contracts, in which employment contracts were made long before employment would commence, via exploding offers that left most applicants with very little ability to compare options.  This kind of market failure afflicted not only the market for new physicians (residents), but also the market for later specialization (as fellows). Consequently, over the years, many specialties have turned to matching for their fellowship positions as well.

  The boxes in brown in the schematic are those that constitute "The Match:" the formulation and submission of the ROLs, and the processing of these into a stable matching of programs to residents.  Congestion is bedeviling the parts in blue.

The boxes colored brown are 'The Match' in which participants formulate and submit rank order lists (ROLs), after which a deferred acceptance algorithm produces a stable matching of applicants to programs, which is accepted by programs and applicants on Match Day. The boxes in blue, the applications and interviews that precede the Match, are presently suffering from some congestion.  Some specialties have been experimenting with signals (loosely modeled on those in the market for new Economics PhDs, but implemented differently by different medical specialties).

The proposal in question was to divide the match into two matches, run sequentially, with the first match only allowing half of the available positions to be filled.  The particular proposal was to do this first for the OB-GYN specialty, thus separating that from the other specialties in an early match, with only half of the OB-GYN positions available early.

This proposal came out of a study funded by the American Medical Association, and it was claimed, without any evidence being offered, that it would solve the current problems facing the transition to residency.  Our paper was written to provide some evidence of the likely effects, by simulating the proposed process using the preferences (ROLs) submitted in previous years.  

The results show that the proposal would largely harm OB-GYN applicants by giving them less preferred positions than they could get in a traditional single match, and that it would create instabilities that would encourage strategic behavior that would likely undermine the successful operation of the match in subsequent years.

Itai Ashlagi, Ephy Love, Jason I. Reminick, Alvin E. Roth; Early vs Single Match in the Transition to Residency: Analysis Using NRMP Data From 2014 to 2021. J Grad Med Educ 1 April 2023; 15 (2): 219–227. doi: https://doi.org/10.4300/JGME-D-22-00177.1

Abstract:

"Background--An Early Result Acceptance Program (ERAP) has been proposed for obstetrics and gynecology (OB/GYN) to address challenges in the transition to residency. However, there are no available data-driven analyses on the effects of ERAP on the residency transition.

"Objective--We used National Resident Matching Program (NRMP) data to simulate the outcomes of ERAP and compare those to what occurred in the Match historically.

"Methods--We simulated ERAP outcomes in OB/GYN, using the de-identified applicant and program rank order lists from 2014 to 2021, and compared them to the actual NRMP Match outcomes. We report outcomes and sensitivity analyses and consider likely behavioral adaptations.

"Results--Fourteen percent of applicants receive a less preferred match under ERAP, while only 8% of applicants receive a more preferred match. Less preferred matches disproportionately affect DOs and international medical graduates (IMGs) compared to US MD seniors. Forty-one percent of programs fill with more preferred sets of applicants, while 24% fill with less preferred sets of applicants. Twelve percent of applicants and 52% of programs are in mutually dissatisfied applicant-program pairs (a pair in which both prefer each other to the match each received). Seventy percent of applicants who receive less preferred matches are part of a mutually dissatisfied pair. In 75% of programs with more preferred outcomes, at least one assigned applicant is part of a mutually dissatisfied pair.

"Conclusions--In this simulation, ERAP fills most OB/GYN positions, but many applicants and programs receive less preferred matches, and disparities increase for DOs and IMGs. ERAP creates mutually dissatisfied applicant-program pairs and problems for mixed-specialty couples, which provides incentives for gamesmanship."



************
I'm hopeful this paper will effectively contribute to the ongoing discussion of how, and how not, to modify the design of the whole process of transition to residency with an aim to fixing the parts that need fixing, without damaging the parts that work well, i.e. while doing no harm. 

(Signaling will likely continue to play a role in this.)



Saturday, January 28, 2023

Signaling in the markets for new doctors

 Signaling of interest is catching on in medical labor markets for residents and fellows.

Here's some material from Thalamus (which describes itself as "Complete GME interview management solution for applicants & programs. Easy, secure, and automated interview scheduling to optimize in-person & virtual recruitment.")

The Ultimate Guide to Preference Signaling for Medical Residency Applicants and Programs 2022-2023.

It all seems to have started with the signaling mechanism we use in Economics.

From Part 1: 

"The Emergence of Preference Signaling:
Preference signaling was first implemented in 2006, as part of the recruitment process for economics graduate students administered through the American Economics Association (AEA). Since then, there have been several useful studies analyzing this process by leading economists at institutions including Harvard and Stanford. These include “Preference Signaling in Matching Markets” and “The Job Market for New Economists: A Market Design Perspective.”

"Of note, one of the authors on the latter article is Dr. Alvin E. Roth, who won the Nobel Prize in Economics for proving certain key attributes of the matching algorithm that is used today by the National Resident Matching Program (NRMP), where Dr. Roth currently serves as a board member. This article has been cited in papers throughout GME that examine preference signaling in specialties including Otolaryngology and Orthopaedic Surgery."
********
Earlier:


Wednesday, November 23, 2022

Improving the transition to (surgical) residency

The transition from medical school to residency is presently troubled by congestion involving (too) many applications and interviews.  It's a subject of considerable discussion in the medical community, sometimes hampered between the parts of the process that proceed the Match, and the Match itself (which is the clearinghouse run by the NRMP that, after all applications and interviews have been processed, solicits rank order lists and turns them into a matching of doctors to residency programs)..  Here's a paper that focuses sensibly on the runup to the Match, even though its title follows the (unfortunately common) practice of calling the whole process the Match.

Designing the “match of the future”: challenges and proposed solutions in the interview and match phase of the UME–GME transition by Sophia K. McKinley, Maria S. Altieri, Olabisi Sheppard, Kimberly Hendershot, Keneeshia Williams, Brigitte K. Smith on behalf of the ASE Graduate Surgical Education Committee, Global Surgical Education - Journal of the Association for Surgical Education : 17 November

Table 1 Challenges and proposed solutions in the surgical resident selection process (click to embiggen)




Friday, October 21, 2022

The past and future of the transition from medical school to residency, in the Journal of Graduate Medical Education, by Williamson, Soane, and Carmody

 The October issue of the Journal of Graduate Medical Education considers the past and future of the transition to residency.

The US Residency Match at 70: What Was, What Is, and What Could Be  by Edwin Williamson, MD; Caroline Soane, BA; J. Bryan Carmody, MD, MPH, J Grad Med Educ (2022) 14 (5): 519–521., https://doi.org/10.4300/JGME-D-22-00248.1

"But while early offers are long gone, the residency selection process now faces a new set of challenges related to the increasing number of applications submitted by contemporary applicants. In 2020 the average US medical school graduate submitted 70 residency applications.9  The average for some specialties is even higher. For instance, in 2022, the average osteopathic medical school applicant in obstetrics and gynecology submitted 85 applications, while US MD applicants in orthopedic surgery submitted 96 applications, and international medical graduates submitted 100 applications each to internal medicine programs.10  This overapplication increases costs for applicants and programs, leads to reliance on convenient screening metrics in applicant evaluation, and does not ultimately improve Match rates.9 "

Tuesday, July 19, 2022

Reducing the costs of preparing for high stakes exams by reporting scores coarsely

 In many countries, national exams serve as the gateway to college admissions and other prizes, and many applicants incur great costs in time and treasure preparing for these exams.  Here's a recent NBER working paper that suggests that reporting the grades in intervals rather than by individual scores has the potential to reduce the costs devoted to exam prep sufficiently to be a Pareto improvement for students, i.e. to make them all better off, even those who obtain the highest grades, if the cost of doing so is sufficiently high.

Pareto Improvements in the Contest for College Admissions by Kala Krishna, Sergey Lychagin, Wojciech Olszewski, Ron Siegel & Chloe Tergiman, NBER WORKING PAPER 30220, DOI 10.3386/w30220, July 2022

Abstract: "College admissions in many countries are based on a centrally administered test. Applicants invest a great deal of resources to improve their performance on the test, and there is growing concern about the large costs associated with these activities. We consider modifying such tests by introducing performance-disclosure policies that pool intervals of performance rankings, and investigate how such policies can improve students’ welfare in a Pareto sense. Pooling affects the equilibrium allocation of studentso colleges, which hurts some students and benefits others, but also affects the effort students exert. We characterize the Pareto frontier of Pareto improving policies, and also identify improvements that are robust to the distribution of college seats.

"We illustrate the potential applicability of our results with an empirical estimation that uses data on college admissions in Turkey. We find that a policy that pools a large fraction of the lowest performing students leads to a Pareto improvement in a contest based on the estimated parameters. We then conduct a laboratory experiment based on the estimated parameters to examine the effect of such pooling on subjects’ behavior. The findings generally support our theoretical predictions. Our work suggests that identifying and introducing Pareto improving performance-disclosure policies may be a feasible and practical way to improve college admissions based on centralized tests."

The paper notes that:

" In many Asian countries, including China, Japan, South Korea, and Taiwan, students attend specialized “cram schools,”1 which focus on improving students’ performance on the tests. This often consists of rote learning, solving a large number of practice problems, and practicing test-taking strategies tailored to the specific test. In other countries, students hire tutors, buy books, and take specialized courses, all geared entirely toward improving their test scores. These activities likely improve students’ performance on the test, but are far less likely to generate substantial long-term improvements in students’ productive human capital. These activities do, however, carry significant costs in terms of time, money, and effort. In South Korea, for example, it is not uncommon for high school students to spend several hours a day in cram schools, and the high stakes competition for college admissions is seen as one of the main causes for the high rates of unhappiness and suicide among teenagers.2 Similar concerns have also been raised in the United States.3"

The paper explains that:

"We are interested in performance-disclosure policies that benefit all students, and refer to such policies as Pareto improving. In particular, we do not need to consider welfare tradeoffs across students. A key finding of our analysis is that Pareto improving policies often exist. This may seem surprising, since a fixed set of college seats implies that a student can be admitted to a better college only if another student is admitted to a worse college. The crucial element that makes Pareto improvements possible is that test preparation is costly. The costs students incur, as well as the resulting college assignment, are determined in equilibrium, and the equilibrium is affected by the performance-disclosure policy. Relative to the baseline contest with no coarsening, introducing a performance-disclosure policy leads to some students being admitted to better colleges; this makes them better off even if they incur higher costs, as long as the cost increase is not too large. Other students are admitted to worse colleges; if they also incur lower costs they are made better off as long as the reduction in the costs is large enough."

**********

I'm reminded of a paper that suggests that the very best students may not pay the highest costs for exam prep:

Feltovich, Nick, Richmond Harbaugh, and Ted To. "Too cool for school? Signalling and countersignalling." RAND Journal of Economics (2002): 630-649.


Sunday, May 8, 2022

Signaling for Otolaryngology residency programs

 Here's a report on the use of signaling for residency positions, from the Oto match.

Pletcher, Steven, Chang, C.W., Thorne, Marc, MD, MPH & Malekzadeh, Sonya. (2022). The Otolaryngology Residency Program Preference Signaling Experience. Academic Medicine, 97, 664-668. https://doi.org/10.1097/ACM.0000000000004441

"The average applicant to Otolaryngology-Head and Neck Surgery in the 2021 residency application cycle applied to more than 50% of otolaryngology programs nationwide, submitting 77 applications, 1 a 34% increase over 5 years. 2 This surge has made it difficult for residency programs to holistically review applications and has limited opportunities for applicants to stand out to programs of particular interest.

...

"A formal preference signaling process provides all applicants with access to a known and stable quantity of signals. Through this process, at the time of application submission, students send a signal to indicate to a defined number of residency programs their particular interest in those programs. Such signals allow students to stand out to their favored programs and allow programs to receive a list of highly interested applicants. To our knowledge, this approach has not been used previously in the residency application process. Yet, articles advocating for signaling exist in the otolaryngology literature, 5-7 and the methodology, rationale, and results of preference signaling for graduates of economics PhD programs applying for faculty positions have been described. 8

"An OPDO working group, comprising the 4 authors, drove the establishment of a signaling process. In the spring of 2020, we held a series of meetings and webinars to engage stakeholders in the development and implementation of a signaling process. We included students, program directors, and specialty societies, such as the Society of University Otolaryngologists and the Association of Academic Departments of Otolaryngology Otolaryngology Chairs Organization. Additional discussions with the Association of American Medical Colleges, the Electronic Residency Application Service (ERAS), and the National Resident Matching Program also took place.

"In hindsight, establishing consensus across stakeholders proved to be the most challenging hurdle to successful implementation of our signaling process. Stakeholders had to accept this change and the inherent risks of implementing a "never before in medicine" process. These discussions, however, also provided a critical opportunity to refine our proposal and create an educational ecosystem that accepted this signaling process.

...

"All otolaryngology residency programs attested to the code of conduct, and none opted out of the signaling process. By October 21, 2020, the date that applications were released to programs, 611 students had submitted applications to otolaryngology residency programs, 559 applicants had submitted a Match list including an otolaryngology program, and 558 applicants had participated in the signaling process. Of 119 non-military otolaryngology residency programs, 118 received at least 1 signal. The number of signals received per program ranged from 0 to 71 with a mean of 22 (standard deviation 17) and a median of 16 

...

"Program directors most commonly reported using signals as a tiebreaker for similar applications and as part of an initial application review algorithm. One program required a signal to offer an applicant an interview.

...

"Applicants reported applying to a mean of 77 programs (standard deviation 21), including their 5 signaled programs. The rate of receiving an interview offer was significantly higher for signaled programs (58%, 670/1,150) compared with both nonsignaled programs (14%, 2,394/16,520; P < .001) and the comparative nonsignal program (23%, 53/230; P < .001; see Figure 2). To assess the impact of signaling across the spectrum of applicant competitiveness, we divided applicants into quartiles based on their overall likelihood of receiving an interview offer. Signals had a significant impact (P < .001) on interview offers across all quartiles (see Figure 3).

...

"The magnitude of signal impact we found likely represents both an increased rate of interview offers from signaled programs and a decreased rate of interview offers from nonsignaled programs. In contrast to our 5 signals, the American Economic Association provides 2 signals for graduates of economics PhD programs applying for faculty positions, suggesting that signal scarcity preserves its value and intent. 8 Decreasing the number of signals would force applicants to narrow their list of programs of primary interest and would disincentivize signaling "dream" programs. By increasing the number of signals, the lack of a signal becomes an indication of disinterest. If enough signals are provided, signaling could have a similar impact on the application process as an application cap. The ideal number of signals then must be explored.

"Our data demonstrated that signaling allowed applicants to influence their likelihood of receiving an interview offer from programs of particular interest. Given the distribution of signals received across programs, we believe that signaling also improved the distribution of interview offers among applicants, which could mitigate interview hoarding. 

...

"While applicants appear to benefit significantly from signaling, they also bear the responsibility of targeting their signals appropriately.

...

"Multiple specialties have expressed interest in adopting a similar program. However, otolaryngology is not representative of all medical or surgical specialties. It is a small, competitive surgical subspecialty with a 63% Match rate and no unmatched residency slots in the 2021 cycle. While the impact of signaling may vary significantly outside of these parameters, we are optimistic that the benefits will carry over to other specialties. Incorporating a signaling option within ERAS would facilitate both wide adoption and further analysis of such a process."

Monday, April 11, 2022

Signaling in medical residencies in the upcoming 2023 season

 One approach that is being adopted in applications for medical residencies is to allow signals of interest.

Here's an announcement from ERAS, the Electronic Residency Application Service® :

Supplemental ERAS® application for the ERAS 2023 cycle

"With the support of their specialty leadership organizations, programs from the following specialties may participate in the supplemental ERAS application for the 2023 ERAS season. "


************

Here's a 2010 paper that includes some of the experience of signaling in the Economics Ph.D. market. We used (and still use) only 2 signals, and we definitely rejected the idea of gold and silver signals...


The Job Market for New Economists: A Market Design Perspective, by Peter Coles, John Cawley, Phillip B. Levine, Muriel Niederle, Alvin E. Roth, and John J. Siegfried (Journal of Economic Perspectives—Volume 24, Number 4—Fall 2010—Pages 187–206)

Friday, August 20, 2021

Preference Signaling and Worker-Firm Matching: Evidence from Interview Auctions, by Laschever and Weinstein

 Here's a recent working paper from the IZA Institute of Labor Economics concerned with the importance of signals of interest in labor market matching:

Preference Signaling and Worker-Firm Matching: Evidence from Interview Auctions, by Ron A. Laschever and Russell Weinstein,  IZA DP No. 14622

Abstract: "We study whether there are improvements in worker-firm matching when employers and applicants can credibly signal their interest in a match. Using a detailed résumé dataset of more than 400 applicants from one university over five years, we analyze a matching process in which firms fill some of their interview slots by invitation and the remainder are filled by an auction. Consistent with the predictions of a signaling model, we find the auction is valuable for less desirable firms trying to hire high desirability applicants. Second, we find evidence that is consistent with the auction benefiting overlooked applicants. Candidates who are less likely to be invited for an interview (e.g., non-U.S. citizens) are hired after having the opportunity to interview through the auction. Among hires, these candidates are more represented among auction winners than invited interviewees, and this difference is more pronounced at more desirable firms. Finally, counterfactual analysis shows the auction increases the number and quality of hires for less desirable firms, and total hires in the market

...

"Auctions for interview slots may address two important frictions in the matching process: uncertainty over applicant quality, and uncertainty over the likelihood that an applicant accepts an offer. Even if employers can successfully identify desirable applicants, there remains the challenge of identifying which candidates are truly interested in the job and would accept an o↵er with high probability. In recent years the cost of job applications has fallen as more postings and applications are online. This
further raises the potential that applicants will have a low likelihood of accepting an offer.
...
"Though not common, there are a few markets in which all applicants have an equal opportunity to credibly signal their preferences for an employer. One example is the American Economic Association (AEA) job signaling mechanism, which allows candidates to send a signal of interest to two departments. Importantly, there is no requirement that employers interview the applicants sending the signal. In contrast, in our setting an employer is compelled to meet with some signaling job seekers.

"A second example, and the focus of this paper, is the auction system used in the market for professional master’s degree students, most commonly MBA students, at many top-ranked programs. These programs allow employers to choose some percentage of the applicants they interview, but require the remainder of the interview slots are allocated through an auction. Typically, firms first invite applicants for interviews, before applicants have had the opportunity to signal. Next, there is an auction for the remaining interview slots, and thus auction participants are students who were not invited for an interview by the firm. Each student is provided with an equal allotment of “bid points,” and the auction winners are guaranteed interviews
with the firm."

Wednesday, April 21, 2021

Signals and interviews in the transition from medical school to residency

Late last year I was interviewed by Dr. Seth Leopold, who is a Professor in the Department of Orthopaedics and Sports Medicine at the University of Washington School of Medicine, and Editor-in-Chief of the journal Clinical Orthopaedics and Related Research.   That interview has just appeared ahead of print on the journal's website: 

A Conversation with … Alvin E. Roth PhD, Economist, Game Theorist, and Nobel Laureate Who Improved the Modern Residency Match  by Leopold, Seth S. MD, Clinical Orthopaedics and Related Research: April 7, 2021 - Publish Ahead of Print - doi: 10.1097/CORR.0000000000001758

Here's one part of our Q&A:

Dr. Leopold:You once commented in a Not the Last Word column in CORR® that the Match might be improved if a bit more room could be made for candidates to send “signals” to programs that indicate particular interest[5]; if you could make one change to the Match right now to make it fairer all around, what would that change be?

Dr. Roth: I don’t yet know enough about the whole pre-Match process of applications and interviews to answer that confidently. I’m hoping to gain access to data that will illuminate more clearly how applications lead to interviews, and how interviews interact with other kinds of information to influence what rank-order lists are submitted by applicants and programs. Some of that process is surely in flux, between the pandemic causing interviews to be conducted remotely and the United States Medical Licensing Examination Step 1 going pass/fail. Signaling is a way to address miscoordination in interviewing (such as whether too many interviews are concentrating on too few candidates), but there are other ways the interview process might be broken that might better be addressed by other tweaks in how interviews are organized.

Dr. Leopold:I believe the study you’re proposing here would find a very attentive audience, both in medical schools and residency programs across the country, especially competitive ones like orthopaedic surgery. Based on other kinds of markets you’ve evaluated—I recognize I’m asking you to speculate—what do you think you might find here?

Dr. Roth: Presently, in at least some specialties, many interviews are conducted for each residency and fellowship position. It could be that interviews play a critical role in allowing programs and applicants to assess each other, regardless of the other information they may have. But it could also be that at least some interviews are being conducted “defensively,” because all the interviews that others are participating in make it hard for each program or applicant to predict how likely any interview will lead to a position being offered and accepted in the Match. So, it is possible that there is “too much” interviewing, in the sense that in perhaps predictable ways, some programs are interviewing some candidates they can virtually never hire, and some candidates they would never want to hire. Conversely, applicants are interviewing for some jobs they have hardly any chance of being offered, and some they sensibly think they won’t need to take. Of course, some things can be predictable even if they can’t be predicted by individual applicants and programs with the information they now have available. It might therefore be possible to suggest institutional reforms that would help reduce the uncertainty in deciding which interviews to offer. That might also reduce the number (and costs) of interviews. (In just such a way, the Match helped solve the problem of uncertainty involved in offers and acceptances, back when offers were exploding.) And there’s a possibility that fewer interviews could make everyone better off in terms of expectations, particularly if participants on both sides of the market will feel a reduced need to do so many interviews if everyone else reduces the number they do. But as you say, until we can look into this carefully, I’m just speculating.

Thursday, February 18, 2021

Economics envy: signalling proposals for the radiation oncology match (to reduce congestion and coordination failure)

 Here's a call to introduce signaling into the resident match for radiation oncologists, explicitly modeled after the signaling used in the job market for economists.

I Need a Sign: The Growing Need for a Signaling Mechanism to Improve the Residency Match

Wesley J. Talcott, MD, MBA , Suzanne B. Evans, MD, MPH, 

Int J Radiation Oncol Biol Phys, Vol. 109, No. 2, pp. 329-331, FEBRUARY 01, 2021, DOI:https://doi.org/10.1016/j.ijrobp.2020.09.008

"Increased attention has been directed toward the radiation oncology interview application process, as the COVID19 crisis threatens to exacerbate its many longstanding inefficiencies.1,  2,  3 A signaling mechanism, by which applicants can send a signal indicating strong interest to a strictly limited number of programs, has been discussed in other specialties to remedy similar inefficiencies4,  5,  6 and will be implemented this application cycle by otolaryngology residency programs. We strongly believe our field should similarly take a leading role on this initiative this application cycle.

Such a system is not novel. A signaling mechanism was introduced in the Economics postdoctoral job market match, which has resembled the Radiation Oncology market in terms of applications submitted per applicant and interview invitations per application. Candidates submitting applications have the option to signal to employers with up to 2 “signals of interest.” These signals are sent through a secure portal associated with the American Economic Association,7 rather than through an extrinsic third party. The instructions for the portal encourage applicants not to send signals to their top 2 programs, but instead to programs who would be likely to factor the signal into their interview invitation decision. 

...

"To improve the interview application process, we recommend granting applicants a strictly limited supply of free virtual “signals of interest” that can be submitted via a monitored online portal. An applicant would be allowed to submit up to 2 signals from this portal to programs of interest. Signals must be limited, as value of these virtual signals comes from their scarcity; strict enforcement of a 2-signal limit means received signals convey credible interest, whereas the absence of an accompanying signal (the case with the vast majority of submitted applications) conveys essentially no information. This system should be free for applicants, to not introduce more financial inequities in this process. The cost to set up the online portal would be minimal given its simplicity, with little overhead once established, and could be shouldered by programs or radiation oncology organizations."

Thursday, January 21, 2021

SAT eliminates subject tests

 The portfolio of standardized tests available to college admissions offices is shrinking (or at least changing)...

The WSJ has the story:

College Board Eliminates SAT Subject Tests--Decision takes effect immediately, while many colleges already made the exams optional   By Melissa Korn and Douglas Belkin

"The College Board is eliminating SAT subject tests, as the pandemic accelerates a push for changes in college admissions.

"The 20 subject tests have been offered for decades in areas including math, English literature, world history and physics, but have fallen out of favor as a requirement for college applications. Between 2016 and 2019, registrations for the test fell by 8%—and dropped sharply last year, as test sites were closed due to the coronavirus pandemic.

...

"Robyn Lady, director of student services at Chantilly High School in Virginia, applauded the latest move. ...

“Anything that moves us closer to simplifying the process for students and removing barriers is a move in the right direction,” she said, adding that she’d like to see all standardized tests eliminated from college admissions. “This is all about equity and access.”

Tuesday, November 3, 2020

Preference Signaling for the Otolaryngology Interview Market

 From the journal The Laryngoscope, a thoughtful description of the growing interest in signaling for medical residency interviewing (which I believe will be implemented for Otolaryngology residency positions in the coming year).

Preference Signaling for the Otolaryngology Interview Market

C.W. David Chang MD  Steven D. Pletcher MD  Marc C. Thorne MD, MPH  Sonya Malekzadeh MD

First published: 06 October 2020 https://doi.org/10.1002/lary.29151

"The impact of the coronavirus disease 2019 pandemic extends beyond patient care and into graduate medical education (GME). The pandemic has created disarray in the residency application process. Visiting rotations and residency interviews—two cornerstones of the application cycle—are gone.

"Just as the pandemic has exposed healthcare disparities in medical care, it also shines a light on inequalities with GME. Even before the pandemic, many residency specialties observed a meteoric rise in the number of applications submitted by each applicant. In 2019, otolaryngology applicants submitted an average of 72 applications, an 80% rise over 15 years.1 This increase drives a cycle of programs receiving more applications and students feeling the need to apply more broadly to maintain competitiveness. Students with monetary resources are better able to mitigate match risk through prolific residency application and by traveling for away rotations to cultivate faculty advocates. Financially disadvantaged applicants may find it more difficult to amass influential social capital.

"With the deluge of applications, applicants are unable to distinguish themselves from the crowd. Such dilution impairs the applicants' abilities to credibly convey interests to programs. Similarly, the program director has a hard time selecting candidates from a pool of excellent applicants for interview.

"The interview is a limited resource. Selection committees often react to this scarcity by declining to interview qualified candidates they think (but do not really know) are unlikely to choose their program and instead interview candidates who they think (but do not really know) are more likely to accept an offer. This approach is inefficient.

"Preference signaling is an intriguing solution. Since 2006, the American Economic Association has operated a signaling service to facilitate job interviews for graduate students. This applicant‐initiated concept aligns goals of interested applicants with programs. Students send signals to up to two employers to indicate their interest in receiving an interview. In reviewing their outcomes, signals were found to increase probability of interview, especially for niche scenarios (nongraduate applicants, applications to liberal arts colleges, and small city locations).2 Signaling has received interest among medical residency specialties as well.3, 4

...

"The Otolaryngology Program Directors Organization (OPDO) Council has worked diligently with stakeholders to incorporate their input throughout the development process. We thank members and leaders of the academic otolaryngology community, including the Association of Academic Departments in Otolaryngology (AADO), the Society of University Otolaryngologists (SUO), and the greater community of program directors for their support. We appreciate the guidance and valuable insight from the Association of American Medical Colleges (AAMC)—specifically the Group on Student Affairs (GSA), the Committee on Student Affairs (COSA), and the Electronic Residency Application Service (ERAS)—along with the National Resident Matching Program (NRMP). We are hopeful that signaling will improve the residency interview selection process by facilitating the successful pairing of applicants with programs."

Wednesday, September 23, 2020

Pandemic disruptions in the market for medical residents

 The coronavirus pandemic and associated lockdowns and limitations have stressed a number of labor markets, including the one for new physicians.  Here's an article from the Journal of Surgical Education that suggests that, in a world of online interviewing, the number of interviews might usefully be capped. They also recommend signalling...

The Case for Capping Residency Interviews

Helen Kang Morgan, MD,*,1 Abigail F. Winkel, MD,† Taylor Standiford, BS,‡ Rodrigo Muñoz, MD,§ Eric A. Strand, MD,║ David A. Marzano, MD,* Tony Ogburn, MD,¶ Carol A. Major, MD,# Susan Cox, MD,⁎⁎ and Maya M. Hammoud, MD, MBA

J Surg Educ. 2020 Sep 14, doi: 10.1016/j.jsurg.2020.08.033 [Epub ahead of print] PMCID: PMC7489264

"As a result of the COVID-19 pandemic, residency programs will make an abrupt shift to virtual interviews in the 2021 residency application cycle.1 ... Medical students, medical schools, and residency programs have needed to react to sudden developments such as cancelled clinical electives, delayed or cancelled United States Medical Licensing Exams (USMLE),2 significant limitations on visiting student elective and sub-internship rotations,3 and changes in Electronic Residency Application Service (ERAS) timelines.4 Given this context, applicants may opt to increase their total number of residency applications as well as interviews accepted and completed, especially since they will no longer be limited by travel and cost deterrents.5, 6, 7 Likewise, residency programs are no longer logistically restrained to configure an applicant's interview schedule on a single day, and will have the ability to schedule interviews throughout multiple days and during non-business hours. In-person interactions provided by traditional interview day experiences have historically weighed heavily in determining mutual compatibility8 , 9; thus, both stakeholder groups will be looking to raise their chances of finding a match, including potentially increasing the number of interviews.

This is particularly troublesome given the current state of residency application processes, "fraught with misaligned stakeholder incentives.10, 11, 12, 13, 14, 15 Although the ratio of positions per applicant is higher now than ever before, the number of applications per applicant have risen.16 , 17 These numbers have increased rapidly in certain specialties, with the mean number of applications per applicant in obstetrics and gynecology (OBGYN) rising from 28 in 2010 to 66 in 2019.18 The consequences of application inflation are numerous and include decreased abilities for residency programs to perform holistic review of applicants with increased reliance on metrics such as USMLE scores. Residency programs also need to devote significant faculty and administrative time for the interview processes.19 The consequence of application inflation that will be of crucial importance this application cycle is the growing awareness that a small percentage of applicants has been receiving a disproportionate percentage of interview offers.20 , 21 In the era of virtual interviews, if these applicants choose to schedule all of their interview offers, there is a real potential for detrimental downstream effects to other applicants. This may also lead to a greater number of unfilled residency spots, with a larger number of programs and applicants required to enter into the Supplemental Offer and Acceptance Program. Given the paucity of data to inform best practices, there is a pressing need to model the potential effects of current application processes and applicant strategies in this disrupted application cycle."
...
"All stakeholders urgently need equitable solutions that address both individual and systems-level problems for this coming application cycle and beyond.30 Capping the number of interviews that an applicant can schedule could remedy 1 pressing flaw in current application interview processes. Implementing caps at the interview scheduling stage is preferable to capping at the application stage given the multiple complexities that must be considered such as DO and IMG status, and overall competitiveness. In addition, exceptions may need to be made for individuals participating in the Couples Match. The potential legal implications of mandatory interview caps are in the infancy of exploration. Capping interviews would likely not violate anti-trust laws given that applicants would still have the choice of where they would like to interview, however these issues would need to be further investigated. New measures such as preference signaling mechanisms30, 31, 32 need to be urgently considered in order for programs to be able to prioritize whom to offer interviews. The use of “tokens” would enable applicants to be able to convey interest to a set number of programs; this has been well-described in graduate PhD economics education literature.33 It will be imperative for “fit” to not become a proxy for decisions guided by unconscious bias,34 but instead, for principles of equity and inclusion to guide change during this time of accelerated change."


HT: Marc Melcher

Friday, December 20, 2019

Like a virgin? Controversy surrounding "virginity tests."

Virginity testing has been in the news.

In the NY Times:
After the Rapper T.I.’s Remarks, N.Y. May Ban ‘Virginity Tests’
Legislation was introduced after the rapper said he subjects his daughter to a yearly hymen exam, sparking outrage on social media.
**********

In the Guardian:
'Now I have to check your hymen': the shocking persistence of virginity tests
In the US, it is still perfectly legal to doctors to perform ‘hymen checks’ as proof of virginity

"a small 2017 study found that of 288 US obstetrician and gynaecologists who were asked, 45 (16%) had been asked at least once to perform virginity testing or virginity “restoration”. Thirteen of those doctors complied.
...
"At present, it is not considered medical malpractice to perform a hymen examination – in fact, it’s completely legal. But one New York assemblywoman is hoping to make a tangible difference by changing the law.

"Michaelle Solages’s bill hopes to take the question completely out of a doctor’s hands. If passed, it will ensure that virginity testing is banned; that any medical professional undertaking such a practice will face losing their license; and that if the examination is performed in the US, whether inside or outside the medical office, it should be constituted as sexual assault.
**************

Here's an old blog post that claims that many young Zulu women in S. Africa are glad to be able to send a signal about their virginity, in defiance of a ban on virginity testing there.

Friday, September 26, 2008

Here's a review of the scientific literature:

Reprod Health. 2017; 14: 61.
 doi: 10.1186/s12978-017-0319-0
Virginity testing: a systematic review
Rose McKeon Olson and Claudia García-Moreno

"Main Results: Seventeen of 1269 identified studies were included. Summary measures could not be computed due to study heterogeneity. Included studies found that hymen examination does not accurately or reliably predict virginity status. In addition, included studies reported that virginity testing could cause physical, psychological, and social harms to the examinee.

"Conclusions: Despite the lack of evidence of medical utility and the potential harms, health professionals in multiple settings continue to practice virginity testing, including when assessing for sexual assault. health professionals must be better informed and medical and other textbooks updated to reflect current medical knowledge. Countries should review their policies and move towards a banning of virginity testing."
***********
And here's an update featuring a story published this morning in the Washington Post--apparently the recently defeated Republican governor of Kentucky hadn't been an avid reader of the scientific literature...

Kentucky’s ex-governor pardoned a child rapist because the 9-year-old victim’s hymen was intact By Antonia Noori Farzan  Dec. 20, 2019

"Already under fire for handing out pardons to relatives of his supporters, Bevin is now facing an onslaught of criticism from medical and forensic experts. Scientists have debunked the notion that inspecting an alleged victim’s hymen can prove whether they were sexually assaulted, and found that most survivors of child sexual abuse do not have any physical damage. George Nichols, an expert in evaluating child abuse who also served as Kentucky’s chief medical examiner for 20 years, told the Courier-Journal on Thursday that Bevin “clearly doesn’t know medicine and anatomy.”

"Bevin, who had a reputation for making controversial and unproven claims during his single term as Kentucky’s governor, was narrowly defeated by Democrat Andy Beshear in November. Before leaving office, he issued 428 pardons, a group that includes multiple convicted murderers and sex offenders, the Courier-Journal reported."

Friday, February 1, 2019

Colleges harvest signals of interest in more ways

Congested markets--those in which there are more potential transactions than can be easily processed--promote signaling, and the search for signals, about which transactions to pursue.  College admissions is a famously congested market, particularly since it became easy for students to submit many applications. So all but the most elite colleges have long searched for signals of "demonstrated interest."  The WSJ has an update on how technology is changing that search.

The Data Colleges Collect on Applicants
To determine ‘demonstrated interest,’ some schools are tracking how quickly prospective students open email and whether they click links  By Douglas Belkin

"Enrollment officers at schools including Seton Hall University, Quinnipiac College and Dickinson College know down to the second when prospective students opened an email from the school, how long they spent reading it and whether they clicked through to any links. Boston University knows if prospective students RSVP’d online to an event—and then didn’t show.
...
"At Seton Hall University, in South Orange, N.J., students receive a score between 1 and 100 that reflects their demonstrated interest, said Alyssa McCloud, vice president of enrollment management. The score includes about 80 variables including how long they spent on the school’s website, whether they opened emails and at what point in high school they started looking on the website (the earlier the better).
...
"In 2017, 37% of 493 schools surveyed by the National Association of College Admission Counseling said they consider demonstrated interest to be of moderate importance—on par with teacher recommendations, class rank and extracurricular activities. It carried less weight than grades, class rigor or board scores.
...
"Colleges also have low-tech means to help determine demonstrated interest. Last year, one third of students who applied to American University either visited its Washington, D.C., campus or attended an information session about the school, said Andrea Felder, assistant vice provost for undergraduate admissions. Two thirds of those admitted took part in either the campus tour or offsite information session.
...
"Mary Hinton, a senior at Dickinson College, benefited from demonstrated interested without knowing it. After she toured Dickinson in high school, she sent a thank-you note to her tour guide, at her mother’s suggestion.

Now a tour-guide herself, Ms. Hinton has learned those notes are forwarded from tour guides to admissions officers. Her advice to prospective students about thank-you notes: “Write them. It just takes a minute and it can make a difference.”

Sunday, August 26, 2018

Signaling and matching in an online labor market, by Horton and Johari

Here's a paper on signaling and matching in a prominent but un-named online labor market that is readily identifiable.

Engineering a Separating Equilibrium

John J. Horton and Ramesh Johari
August 14, 2018

Abstract: This  paper  explores  whether  platform-created  signaling opportunities can move designed markets to more desirable equilibria.  In a large on-line  labor  market,  buyers  were  given  the  opportunity  to  signal  their relative  preferences  over  price  and  quality.   The  intervention  caused substantial sorting by sellers to buyers of the right “type.”  However, sellers clearly tailored their bids to the type of buyer they faced, bidding up against sellers with a high revealed willingness to pay.  Despite this “markup,” a separating equilibrium was sustained over time, suggesting buyers found revelation incentive compatible.  We find evidence that informative signaling improved matching efficiency and match quality.

"The signaling opportunity was simple:  when posting a job opening,  employers selected one of three “tiers” to describe the kinds of applicants they were most interested in:  (1) Entry level:  “I am looking for [workers] with the lowest rates.”;  (2) Intermediate:  “I am looking for a mix of experience and value.”; (3) Expert:  “I am willing to pay higher rates for the most experienced [workers].”  We refer to these tiers as “low,” “medium,” and “high,” respectively.   When  the  signaling  opportunity  was  introduced  market-wide  (which occurred after an experimental period), the tier choice was revealed publicly to all job-seeking workers."
*************