May 7, 1999

I am writing in response to Dr. Diana Dell's article in the April 2 issue justifying and demanding reform in the ABPN examination. As I read it, Dr. Dell, on behalf of residents and early career psychiatrists, objects to the pass rate of 60.7 percent as prima facie too low ( that is, if the exam were any good, more people would pass it) and also because she found the written exam to be irrelevant to the clinical practice of psychiatry (that is, as a resident-in-training I know what is and is not relevant to the clinical practice of psychiatry).

As a solution, Dr. Dell proposes that the ABPN publish program-specific data so that the residents will know whether their program is doing a good job and make the ABPN more accountable to the public and its constituents, asserting as if it were self-evident (because it is emotionally appealing) that the ABPN operates without any accountability, limitations, concern, or consideration for its serious responsibilities, including to the public, the profession, and those who do sacrifice time and money to take the exam.

In regard to the latter, Dr. Dell really goes beyond the bounds of appropriate discourse to suggest that the ABPN may preserve the low pass rate to make more money on the retakes. At no point does Dr. Dell suggest that the problem may lie in the candidate who fails rather than in the exam and/or in the training program, although I would grant that training programs often should share in the responsibility for a failing candidate.

There is always value in self-examination and justified change, and there have been many changes in the process and structure of the boards over the years. I served as an examiner for over 20 years and over 40 exams, for many of those as a senior examiner; I also have been a training director and a chair of a psychiatry department, and I have had some experience with APA, as well as the Residency Review Committee for Psychiatry as a member for nine years. I consider myself somewhat knowledgeable about the issues Dr. Dell raises, and would address the following points:

1. The low pass rate is not of itself any indication of the validity of the test, anymore than the 92 percent pass rate, which Dr. Dell cites for family practice, is a priori an indication of laxity or ease of passage. However, I would be more concerned about the latter exam than the former if one purpose of the exam is to insure to the public that the physician so certified is safe and competent.



2. The ABPN itself, through its process of developing questions and through its instructions to and supervision of examiners, strives to make the written and oral as equitable and fair as possible; in fact, many examiners have complained that the exam sets the bar too low rather than too high.



3. Dr. Dell does not mention-and it has been a sensitive topic to discuss openly-the fact that psychiatry has the highest percentage of international medical graduates (IMGs) in residency (and has had so for several years) and that IMGs are disproportionately represented in the fail rate. That is an ongoing and serious problem for our training programs, our residents, our profession, and the public. This should not reflect adversely on any individual or nationality of IMGs, but it also is a fact that medical education is not the same all over the world, and IMGs represent a broader range of quality of medical education than do American medical graduates (AMGs). This should not be misconstrued to imply that AMGs come without variety or range of excellence.



4. So, do you penalize the programs that have a higher percentage of exam failures if that program also has a higher percentage of IMGs; or do you try to help the program to do better; or do you try to close the program, as the RRC has tried to do on many occasions against all the safeguards of due process and appeals; or do you publish the data so that the low-pass program will have even more difficulty recruiting residents, even when the problem is not so much in the curriculum or in the supervision? The problem is multifaceted, including those programs that graduate a resident (both IMGs as well as AMGs) who may be sub par and will have difficulty passing the boards.



5. The publication of program-specific pass/fail rates is really not the answer, however appealing it seems; it is indeed likely to cause more problems than solutions.

Whatever improvements may or may not need to be made in the content of the written exam, the more important issues that need to be addressed are in the accreditation process for training programs conducted by the Residency Review Committee, composed of appointed representatives from APA, the ABPN, and the AMA. This committee does receive program-specific data (a struggle with the ABPN resolved several years ago) and can and does use these data in its surveys and in its accreditation decisions. There are perhaps many training programs that should be closed, but this is a lengthy and tedious process subject to many levels of appeal and review. I would urge the members-in-training and early career psychiatrists to invest their efforts for reform in dialogue with the RRC regarding the process and structure by which accreditation standards and review are determined and acted out. That is where a real understanding of the problems and genuine struggle with reality-based conflicts of interest will be encountered.

JERRY M. WIENER, M.D.
Washington, D.C.

Dr. Wiener is a former president of APA.

As a recently board-certified psychiatrist, I have to applaud Dr. Diana Dell's bold article about the ABPN exam in the April 2 issue. This certification process needs to become accountable. Totally subjective grading under the guise of "privacy protection" allows the ABPN to continue its autocratic rigidity and lack of accountability. Will the recertification exam be as poorly contrived? Do APA and the AMA know or care?

If passing the boards suggests "competency," then presumably psychiatrists who are "less than competent" are being allowed to practice. Until "organized" psychiatry faces the burden of meaningfully approximating competency and engages in active self-monitoring, we allow the door open for other entities to define our boundaries-entities that answer to no one, like the ABPN, are the most toxic of all.

ANNE MILLER, M.D.
Wadsworth, Ohio

I write to applaud the article by Dr. Diana Dell on the need for reform in the ABPN exam process. My colleagues and I from Johns Hopkins have had many similar gripes about both parts of the exam, but especially Part II.

When we took the written boards in 1996, we agreed that there was a paucity of clinically relevant material. The test seemed very unbalanced. In fact, there must have been 10 questions about EEG changes in alcoholic patients (one would have been O.K.). Perhaps we just had a peculiar test. Fortunately, we all passed, and, it is hoped, general knowledge about issues relevant to psychiatry contributed to our success.

With regard to the Part II exam, many of my colleagues and I have remarked that, though we know each other well and are familiar with our levels of clinical competence, we are surprised by who does not pass the test. It seems clear, however, that those who pass tend to have practiced doing brief, cursory exams ("mock orals") before the test. Thus, passing seems not so much a function of clinical competence as it is a function of preparation specifically for passing Part II. A few ABPN studies have shown that the reliability of this test, when measured, is fair (this is with two raters measuring the performance of a candidate on the same occasion, with the same patient). Others have publicly criticized the methods of these studies on methodologic grounds, pointing out how procedures used in the process and analysis could falsely elevate reliability estimates.

Since we cannot achieve test validity without first developing reliability, how can we expect a test with only fair interrater (not test-retest) reliability to classify candidates validly as adequate or inadequate clinicians? Even with adequate reliability, validity is not assured. I would like to see evidence that the Part II exam actually passes most good clinicians and fails almost all poor ones. To my knowledge, such a study has not been adequately reported for the general psychiatry examination (though one such study was reported for the child and adolescent psychiatry exam, and another was preliminarily reported for the general psychiatry exam).

Dr. Dell's call for reform struck a sympathetic chord in me. As I've reviewed the literature, I have come to realize that my thoughts on the oral exam are by no means novel. Remedying any current problems with the content of the written exam seems relatively easy. Though it may be more difficult to improve (or replace) the oral exam, it is clearly a worthwhile enterprise, given the economic and emotional consequences of failing this test.

JOE BIENVENU, M.D.
Baltimore, Md.

The list of references for this letter may be obtained by calling (202) 682-6137.