ACFD – CDA Admissions Report – May 2014

Download Complete Report

CDA-ACFD Admissions Report Final

EXECUTIVE SUMMARY

 Project aims

Given the mandate for the working group, we identified the following aims as a means to direct our work. We aimed to:

  • Describe the admissions processes used for the dentistry programs in each dental school in Canada;
  • Describe the problems and potential solutions and issues people working in Canadian dental schools have identified in relation to the current admissions processes for dentistry programs;
  • Describe the problems and other issues people in the broader dental profession identify in relation to admissions to dentistry programs in Canada;
  • Describe the research literature concerning the validity and reliability of various instruments and processes used in admissions to dentistry programs;
  • Where relevant, compare this literature with that from medicine and nursing and veterinary medicine;
  • Draw conclusions concerning the quality of the admissions instruments and processes currently used for dentistry programs in Canada; and
  • Make recommendations concerning possible future means to ensure high quality admissions processes are used for dentistry programs in Canada.

With these aims in mind, working group members decided to use a mixed methods approach to address them. The approach included a review of the relevant literature and interviews with admissions personnel at all Canadian dental schools, plus interviews with some national dental organizations in Canada and the USA.

The literature review on validity of admissions processes

After reviewing the literature and evaluating potential assessment tools used during the dental school admission process, we found that some DAT components (mainly DAT-AA [academic average], DAT-QR [quantitative reasoning] and DAT-RC [reading comprehension]) and Pre-dental Overall GPA and Science GPA scores represent the best predictors of academic performance of dental students. Also they have some correlation with clinical and board examination performance, although the strength of these relationships is only weak to moderate. Some questions remain concerning the concurrent validity of the DAT-PAT (perceptual ability test), since its results were not compared with those of any other forms of perceptual ability assessment. Our results also suggest that combining cognitive assessment tools together but also with non-cognitive assessment tools considerably increases their predictive validity. The debate still remains regarding which non-cognitive tools to use.

Interviews remain a popular way to assess non-cognitive and personality traits. Our findings show that increasing the structure of the interview, whether it is through a structured or MMI (Multiple Mini Interview) format, drastically increases its reliability and validity. As for comparing MMI and structured interviews, our findings seem to indicate that MMI have a slightly higher reliability, although more research is needed to be able to arrive at more definite results.

Manual dexterity test were found to have null to weak predictive validity on student performance. That being said, manual dexterity may be utilized as a screening tool in admissions and has been shown in the past to reduce attrition rates in dental school.

With respect to the quality of the validity testing for admissions processes, many aspects have to be investigated. Yet, most of the reviewed articles only focused on predictive and convergent/divergent validity, and to a lesser extent, on concurrent validity. Face, construct and content validities of the selection tools were only assessed in the case of MMI. Concerning the reliability, this aspect was only addressed by a few studies pertaining to manual dexterity tests (tweezers, computer-assisted dental simulation) and structured interviews. Therefore, based on this review, there are still many areas that need to be explored before any well-informed statement upon the validity and reliability of selection tools can be made. As for other limits of the articles reviewed, most of the results were based on correlations and/or descriptive statistical analyses, so inferences and conclusions can only be very limited.

When it came to achieving greater student diversity, our findings suggest that students of diverse backgrounds (e.g. rural or racial minorities) are not disadvantaged by the selection tools currently used in the dental admission processes. Therefore, other initiatives such as under-represented minorities recruitment and pipeline-type programs must be implemented if universities want to achieve greater student diversity. Our literature review suggested different possible avenues to achieve this goal.

The results of interviews with stakeholders

Most of the Canadian admissions officers agreed that there is value in having a battery of national, standardized admissions tests to choose from to assess cognitive ability, non-cognitive attributes and psychomotor skills. Used in conjunction with the GPA, a national standardized series of tests would address the concerns raised about GPA. These included: i) grade inflation in some courses within an institution; ii) grade variation by different instructors in the same course in a given institution; iii) variation in GPAs across institutions; and iv) grade variation across programs (e.g. Engineering versus a General Science program)

On the other hand, the Admissions Officers commented that the current test components of the Canadian DAT have some problems that need to be addressed:

  • The Survey of Natural Sciences does not appear to be at the appropriate level to assess the CEGEP (Collège d’enseignement général et professionnel or General and Vocational College) students that apply to the three Quebec dental schools. The standardized test battery that provides the benefits stated above are offset if all ten dental schools cannot use them;
  • Another important problem is that the Reading Comprehension Test is only available in English. A national test battery must be available and accessible for students applying at all ten dental schools in both English and French;
  • The validity and reliability of the DAT test components are not being evaluated at all by the current CDA Admissions Committee. The only current statistical data available is from the ADA Department of Testing Services. Since the Canadian and US DAT test components are different, no analysis is being performed on the Manual Dexterity Test (DAT-CD).

Concern was expressed regarding the limited mandate of the CDA Dental Aptitude Test Sub-Committee. The CDA develops and makes available the DAT and CDA Structured Interview to the dental schools. It has a process to grade the Manual Dexterity Test. However, assessment of the validity and reliability of these tests through the CDA does not occur. Comments from the admissions officers indicated that they tended to attempt to assess various admissions tools at a local level (Faculty Admissions Committee level). This is a direct result of the limitations in the mandate and funding of the current CDA Dental Aptitude Test Sub-Committee. Assessment of non-cognitive attributes in the admissions process was desired by all of the admissions officers. In the absence of a national Committee with the mandate to assess new admissions tools to assess non-cognitive attributes among others, individual schools tend to work in a vacuum. This is not cost-effective or efficient, and does not allow for the collection of data to improve research on admission tool use.

Recommendations

Based on the findings of the literature review and interviews, the working group makes the following recommendations:

  1. The current CDA Dental Aptitude Test Sub-Committee should be replaced by a national Admissions Committee jointly administered by the CDA and the ACFD. The membership of the new Admissions Committee should include:
  • admissions officers from each of the 10 dental schools
  • individuals with expertise in admissions and relevant research
  • individuals with expertise in assessing the validity and reliability of admissions tools
  • individuals with expertise in generating items, scenarios and elements of non-cognitive tests (e.g. MMIs and structured interviews)
  • administrative support
  1. The mandate of the new national Admissions Committee needs to be broadened to include the following functions:
  • Development of guidelines concerning overall student selection and admissions processes (i.e. beyond simply oversight of the DAT);
  • Development of guidelines on the use of specific tools and processes to ensure they are used appropriately e.g. cut-off scores, use for screening etc.;
  • Training on the use of admissions tools;
  • Development of elements of tests and processes (e.g. questions for structured interviews or scenarios for MMI’s); and
  • Oversight of evaluating the validity of admissions tools and processes.
  1. The new national Admissions Committee needs to be appropriately funded to enable the performance of the expanded mandate outlined above.
  2. All admissions tools that the new national Admissions Committee recommends to the schools for consideration in their admissions processes:
  • must be available in both English and French
  • must be at an appropriate academic level to be able to be administered to all applicants to dental programs in Canada, specifically to have a level playing field for University-based and CEGEP-based applicants
  1. Efforts to investigate validation of tests should focus on those showing promise:
  • DAT-AA (Academic Average), DAT-QR (Quantitative Reasoning), DAT-RC (Reading Comprehension);
  • Overall predental GPA, Science GPA;
  • MMI, structured interviews; and
  • Combinations
  1. As the DAT-CD (Manual Dexterity Test) is currently being used by seven of the ten dental schools in Canada, evaluation of the validity and reliability of this admission test component needs to be undertaken, including its most appropriate use.
  2. Efforts should be made by dental schools to focus their admissions processes on tools with the strongest evidence to support them, while ceasing the use of approaches that have little evidence to support them and/or with evidence that shows the tools are not effective in the admissions process.
  3. Efforts should be focused on the recruitment of appropriate candidates as well as the processes used to select them.