Examinations are an opportunity for students to demonstrate their knowledge of the topics being assessed. Like other forms of assessment, the design of examinations needs to be consistent with the UTS Principles of Assessment and allow students to demonstrate their knowledge and skills in a fair and equitable way. Writing fair and equitable exam questions takes time, but is essential if we are to provide meaningful, practice-oriented tasks for students to display their knowledge and abilities related to the subject learning objectives.
Considerations
- Online exams should have approximately the same workload as paper-based exams
- Online exams should cover about the same number of concepts as previous exams
- Online exam questions are likely to be slightly more complex than in-person exam questions and therefore require slightly more reading time than paper-based exams
- Students can usually type faster than they can hand-write and typed answers online are likely to be slightly longer than paper-based exams – clear word limit guidelines for the answers may be necessary to manage marking workloads
Question types
The subject learning objectives provide a guide to the choice of the type of question you will ask in an examination. Subject learning objectives that require the students to define, list, name or identify, suggest that students need to select from a collection of known responses. Selected response questions involve students choosing the correct response or supplying a word or short phrase to answer a question. Examples are multiple choice, multiple answer, true-false, matching or fill-in-the-blank questions.
Subject learning objectives that expect the student to apply, analyse, compare, structure or evaluate suggest that students need to construct an original answer to the question, either as a short written answer or longer essay responses, or as a performance test. With careful writing it is possible to construct effective selected response questions that test the student’s ability to calculate, analyse or evaluate.
Structuring your questions
Structure your questions around set of facts presented in the form of a problem, scenario or case study.
- Problem-based questions present relevant qualitative and quantitative data and requires students to use the information provided to solve a problem or calculate an answer.
- Scenario questions require the additional interpretative skill of students identifying the relevant information from a fictitious situation allowing for a range of confounding variables to be built into the question.
- Case-based questions provide a realistic context constructed from actual examples of situations described in the question.
Devise questions that require students to apply and make use of information through analysis or evaluation.
- Analysis questions ask the student to examine a situation in detail and identify motives and causes, make inferences and find evidence to support their conclusion.
- Evaluation questions ask the student to make judgments about the information and justify a position or decision.
Avoid knowledge recall questions that have answers that can easily be found online.
Online multiple choice questions
The most commonly used selected response item is a multiple-choice question that consists of a stem, a correct or most appropriate answer and distractors. The stem presents the problem or context of the question that students need to answer and should include all factual information needed to answer the question. In most cases the stem is written but can contain other material such as graphs, diagrams or sets of results.
A well-written stem is a complete statement that can be answered without looking at the options. Stems should be clear, specific and stated in a positive form. Understanding the stem should not be a test of comprehension rather a test of the material on which the question is based.
The options need to be written in a way that minimises guessing without clues to the correct answer or clues which allow a student to discard distracters even if little is known of the material under test. The correct answer should be independent of its position in the list of distractors. This allows randomisation of the distractors and a level of personalisation of question responses. This excludes the use of ‘All of the above’ ‘ and ‘None of the above’ as a distractor.
All distractors should be plausible and mutually exclusive. Ideally, each distractor would represent a common student misunderstanding of the concept being examined.
Testing the test
One of the advantages of using online multiple-choice questions is the statistics on the test is easy to obtain and will help determine the quality of the question. Item analysis tools should be used after each online MCQ exam to find out which questions were rated as difficult, whether the top 25% of students did well in all questions and whether some distractors were ineffective in their role of presenting plausible responses to the students.
This information can be used to refine or delete any ineffectual exam questions. Items with acceptable discrimination and facility can be stored for future use. In this way an item bank of high quality questions can be built up over time and questions selected randomly for more personalised tests.