A Good Test vs a Bad Test, or How NOT to Make a Knowledge Assessment Test
It is difficult to make a good knowledge assessment test. I faced this problem back in 2003. We started a distance learning system (DLS) at Vinnytsia National Technical University and began to accumulate a database of tests for various courses. It is impossible to say that all of them were perfect. All teachers, including experienced ones, made mistakes that turned the tests ineffective.
Later, we created our learning platform – Collaborator. This is an LMS for business. And now, when it serves more than fifty companies in Ukraine, I again see similar errors in test questions.
It might seem that making a good test is impossible without solid methodological skills and pedagogical experience. But in fact, common sense, a little time and some simple tricks are enough to receive a good result. I’ll tell you about them.
What is given:
We need to clarify the conditions of our task.
- This is a knowledge assessment test. Such a test determines the level of understanding or confirms the assimilation of information. This is a measuring tool. Its result is a numerical value, the number of points scored.
- We consider only the test with the “closed” type of questions, where you need to make a choice from the proposed options, and not write something of your own. These tests are very popular in the LMS, because the answers to them are checked by the algorithm, i.e. determining the result does not require the participation of a human teacher.
- We accept that there are enough questions for the statistical reliability of the result in the test (at least more than 100 questions). The number of questions sufficient for such reliability is a separate topic for discussion. Here we focus on the quality of test questions.
Solution:
What we should remember when making a test
- Any test question can be asked differently.
- Any knowledge assessment test can be reduced to simple types of questions, such as choosing one or more from many options. Complicating without urgent need is evil.
- We use infostyle for the questions and answer options. If you are not yet familiar with the book Write and Reduce: How to Create a Strong Text, then it’s high time to do it. I recommend to know and apply infostyle when formulating questions. This will make them understandable and reduce stress in test takers.
1. A question should be a question
Do not make your employees to complete unfinished sentences or fill in ragged phrases. They cannot immediately understand either what you want from them, or the meaning and purpose of such a question in the test. They have to re-read all the answers to formulate in the mind the question they will have to answer. Why not just ask the question directly?
Example:
| Wrong | Right |
A feature of the M2 Eco tonometer is:
|
Which of the following are the characteristics of the M2 Eco tonometer?
|
2. Start the question with the main point
Immediately draw the attention of the test taker to what is required of him. Start with the essence of the issue, with the main thing. Follow the example of the English language. A correctly constructed sentence begins with the subject and predicate (who does what). Only then all other details are added.
Example:
| Wrong | Right |
| When selling an Ala-Dofinua special dish, is a branded wet tissue added? | Is it necessary to add a branded wet tissue when selling a special dish “Ala-Dofinua”? |
| An employee can be removed from work by the decision of the employer, drawn up in the form of an Order, in what case? | In what case can an employee be removed from work by the decision of the employer, drawn up in the form of an Order? |
3. Use simple and short phrases
Long sentences are like a ball of thread. One has to untangle all the commas, pronouns and parts of speech in order to understand the meaning of the question. Simplify your questions.
Example:
| Wrong | Right |
| The patient complains of a sore throat, fever, weakness. On examination, the tonsils are covered with abundant light plaque. What pathogen causes sore throat? | What pathogen causes tonsillitis (tonsils covered with abundant light coating)? |
| Put in order the sequence of actions in order to find information about the patient and get to his medical appointments, if you have the number of his outpatient card. | Set the correct search sequence for the patient’s medical appointments by the number of his outpatient card. |
4. “Childish” and “idiotic” questions
This is a “casual” terminology. Questions with an obvious answer are called “childish”, and questions with an impossible answer are called “idiotic”. Both options are not allowed in knowledge assessment tests. They only distort the test result, which leads to an incorrect assessment of the employee.
I must say that “childish” questions are not necessarily something bad. In the case of self-control, they are appropriate. Obvious answers help the authors of training courses to highlight the main and significant information, to focus the listener’s attention on it. But if the “childish” question does not concern the essential aspects of knowledge, it is useless.
Many LMS have tools to detect simple and complex questions. Such questions should be reformulated. Find those to which all employees answer correctly in more than 95% of cases – these are “childish” questions. Questions which receive less than 5% of correct answers can be considered “idiotic”. Remember that this works if all selected testing cases were carried out under the same conditions and there were many of them – at least about a hundred or more.
Example:
| Wrong | Right |
How to minimize the formation of limescale on the windows of shower stalls?
This is a childish question, because the correct answer – the Anti-plaque option – is obvious and easily determined by the exclusion method. |
Which of the following minimizes the formation of limescale on the windows of shower stalls?
|
5. Use believable distractors
Wrong answer options – distractors – should be believable. Otherwise, they can be instantly recognized. A question with such answers is useless both for self-testing and in the certification test.
Moreover, such answer options reveal the incompetence and haste of the test compiler. They often bring a smile to those employees who are at least a little in the know of the subject.
Making the right distractors is a separate topic for discussion. You can read about it briefly on Wikipedia.
Example:
| Wrong | Right |
What action of the manager should be the first one in the event of a fire in a room with people?
|
What action of the manager should be the first one in the event of a fire in a room with people?
|
6. Remove “all answers are correct” and “there are no correct answers”
Firstly, using this approach, we play on the nerves of users and lose their time.
What do you feel when you see the option “all answers are correct”? Doubt? Even a professional begins to doubt whether he understood everything correctly and starts to re-read the question and answer options.
Secondly, this answer option encourages you to choose it or vice versa – not to choose it. In psychology, this is called the “extreme response set-up” trap. The objectivity of the answer to the question is lost.
Thirdly, the use of “no correct answers” causes only confusion and “a sense of trick”. The obvious goal of such questions is to make the test taker fail, and not to determine his level of knowledge.
Remake the question by choosing several answers and add one or two believable distractors. Otherwise, break it down into several questions.
Example:
| Wrong | Right |
An employee can be removed from work by the decision of the employer, drawn up in the form of an Order, in the case of?
|
In what cases can an employee be removed from work by the decision of the employer drawn up in the form of an Order?
|
Conclusion
These are the most common cases of irrational formation of test questions. If you have experience working with problem tests, please share, write your examples and what you did with them in the comments.
If you have the time and desire to thoroughly understand the methods of developing pedagogical tests, you should refer to the science of Testology. Here is what will help you with this:
- Articles and works of Avanesov V.S. – His works are devoted to pedagogical testing and this is one of the best ways to understand the theory of tests.
- Avanesov V.S. Composition of Testing Tasks. Training book. 3rd edition added. Moscow, Testing Center, 2002. 240 p. – The recommendations collected in this publication remain relevant until now.
- The Secrets of Making Tests – A good article on how to make tests on real cases of the banking sector.
- Writing Good Multiple Choice Test Questions. – A 2013 article on compiling multiple-answer tests and ways to create the right distractors.
- Bulah I., Mruha M. Making Qualitative Tests. Tutorial.– Kyiv: Meister-class, 2006. 160 p. – A manual on pedagogical testology, the presentation style is academic, but there are many practical examples.
- A. Shmelev. Practical Testology – recommended by Igor Beh, contains many good examples.
Stay connected with us on Facebook and Linkedin!
Subscribe to our e-learning newsletter, get a discount on LMS Collaborator promo code: LMS2020-05
