ChatGPT Generated Otorhinolaryngology Multiple-Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments.

Details

Serval ID
serval:BIB_57D87EE1F6C3
Type
Article: article from journal or magazin.
Collection
Publications
Institution
Title
ChatGPT Generated Otorhinolaryngology Multiple-Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments.
Journal
OTO open
Author(s)
Lotto C., Sheppard S.C., Anschuetz W., Stricker D., Molinari G., Huwendiek S., Anschuetz L.
ISSN
2473-974X (Electronic)
ISSN-L
2473-974X
Publication state
Published
Issued date
2024
Peer-reviewed
Oui
Volume
8
Number
3
Pages
e70018
Language
english
Notes
Publication types: Journal Article
Publication Status: epublish
Abstract
To explore Chat Generative Pretrained Transformer's (ChatGPT's) capability to create multiple-choice questions about otorhinolaryngology (ORL).
Experimental question generation and exam simulation.
Tertiary academic center.
ChatGPT 3.5 was prompted: "Can you please create a challenging 20-question multiple-choice questionnaire about clinical cases in otolaryngology, offering five answer options?." The generated questionnaire was sent to medical students, residents, and consultants. Questions were investigated regarding quality criteria. Answers were anonymized and the resulting data was analyzed in terms of difficulty and internal consistency.
ChatGPT 3.5 generated 20 exam questions of which 1 question was considered off-topic, 3 questions had a false answer, and 3 questions had multiple correct answers. Subspecialty theme repartition was as follows: 5 questions were on otology, 5 about rhinology, and 10 questions addressed head and neck. The qualities of focus and relevance were good while the vignette and distractor qualities were low. The level of difficulty was suitable for undergraduate medical students (n = 24), but too easy for residents (n = 30) or consultants (n = 10) in ORL. Cronbach's α was highest (.69) with 15 selected questions using students' results.
ChatGPT 3.5 is able to generate grammatically correct simple ORL multiple choice questions for a medical student level. However, the overall quality of the questions was average, needing thorough review and revision by a medical expert to ensure suitability in future exams.
Keywords
ChatGPT, artificial intelligence, exam, large language model, multiple choice question, otolaryngology
Pubmed
Open Access
Yes
Create date
30/09/2024 15:25
Last modification date
01/10/2024 7:07
Usage data