D2.Psy.5.9-12: Validity, Reliability and Data Analysis in Psychology

Overview

Lesson Procedure

60-70 minutes of class time across 5 phases.

1. Warm-Up5-10 minutes
2. Direct Instruction20 minutes
3. Guided Practice15 minutes
4. Independent Practice15 minutes
5. Closure5-10 minutes

1. Warm-Up5-10 minutes

   Warm-Up Prompt

   "A classroom thermometer always reads exactly 2 degrees higher than the real temperature. Is it a good thermometer? What would go wrong, and what would still work, if you used it to track the room for a month?"

   Give students two minutes to write, then collect answers on the board in two columns: "what goes wrong" and "what still works." Students usually notice that every reading is too high. Fewer notice that a change from one day to the next is still correct, because the same 2 degrees are added to both readings. Keep both columns visible: they become the difference between systematic error and a usable comparison in Direct Instruction.

2. Direct Instruction20 minutes

   Every number in a psychology study comes from an observation or a measurement, and data analysis can only be as good as those numbers. Two questions come first. Is the measure reliable: would it give the same result again? Is it valid: does it capture the thing it claims to capture? The table gives the working vocabulary.

   Reliability and validity: the working vocabulary (everyday examples written for this page)

   Term	What it means	Everyday example
   Reliability	Consistency: the measure gives the same result when the same thing is measured again under the same conditions	A kitchen scale shows 500 g every time the same bag of rice is weighed
   Test-retest reliability	Scores stay similar when the same people are measured twice, some time apart	A typing test gives each student nearly the same speed on Monday and on Friday
   Inter-rater reliability	Two or more observers who code the same behavior independently give the same codes	Two referees watching the same play both call it a foul
   Validity	Accuracy of meaning: the measure or study captures what it claims to capture	A driving test that includes actual driving, not only a written quiz
   Construct validity	The measure fits the concept it is named after	Counting words written in two minutes is a poor measure of "creativity"
   Criterion validity	Scores relate to an outside standard they should predict or match	Scores on a fitness test match how far students actually run in 12 minutes
   Internal validity	The design rules out other explanations for a result	Both groups study in the same quiet room, so noise cannot explain a difference
   External validity	The result applies beyond the people and setting studied	A finding from one swim team may not hold for all teenagers
   Random error	Unpredictable ups and downs that differ from one measurement to the next	A stopwatch started a little early on some trials and a little late on others
   Systematic error (bias)	Error that pushes results in the same direction every time	A thermometer that always reads 2 degrees high
   Blind procedure	The person observing or recording does not know the condition or the expected answer	A judge tastes two cookies without knowing which recipe is which

   How the two ideas reach the data analysis. Random error (low reliability) adds noise. Scores spread out more, real differences between groups become harder to detect, and correlations come out weaker than the true relationship. Taking more measurements and averaging them reduces random error. Systematic error (low validity) is different: it shifts or distorts results the same way every time, so averaging more data does not remove it. A large sample measured with a biased tool gives a precise estimate of the wrong thing. That is why reliability is necessary for validity but not enough: a measure must be consistent before it can be accurate, yet a consistent measure can still be consistently wrong.

   The case. In 1904 a horse called Hans drew crowds in Berlin. His owner, Mr. von Osten, a former mathematics teacher, claimed that Hans could read, count and add, answering by tapping his hoof. Oskar Pfungst, a co-worker of the psychologist Carl Stumpf at the Psychological Institute in Berlin, tested the claim (Pfungst, 1911, chapter II). Read Source 1 with the class.

   1With regard to the number of experiments and their performance, the following precautions were observed. A sufficiently large number of tests was made in each series in order to obviate the possibility of the contention that the horse's errors were due to chance. The conditions of experimentation were such that the further contention that he happened to be tired or otherwise indisposed, whenever the reactions seemed to be inadequate, could not be offered. The possibility of confusing the horse by means of unwonted conditions also had to be avoided. For this reason it was necessary to alternate the trial in which procedure was with the knowledge of the answer on the part of the questioner, with the trial in which the procedure was without such knowledge. Such precautions had hitherto been neglected, and therefore those negative results which had been occasionally obtained in single trials, could not claim objective validity, even though the persons making the tests were subjectively convinced.

   [...]

   2The following tests were made in which the method was such that when the problem was presented to the horse, the correct solution was known to none of those present, least of all to the questioner. This method we shall designate in the following report as "procedure without knowledge" whereas we shall call the method in which the answer was known to the questioner, "procedure with knowledge".

   3In order to discover if the horse could read numbers, a series of cards on which numerals were blazoned, were exposed to the horse's view in such a way that none of those present was able to see them, and the horse was asked to tap the numbers as they were shown. This experiment was repeated at different times and in all there were 49 tests in which procedure was without knowledge, and 42 in which procedure was with knowledge. In the case of the former there were 8% correct responses, whereas in the case of the latter 98% of the answers were right. [...]

   [...]

   4Whenever the questioner knew the solution, nearly all of the horse's answers were correct; but when the answers were unknown to the questioner, the horse's responses were, with only a few exceptions, quite unsuccessful. Since the few exceptional cases must be regarded as fortuitous, the conclusion is warranted that the horse was unable to read numerals without assistance.

   [...]

   5The horse's reputed aptitude in computation was tested in the following way. Mr. von Osten whispered a number in the horse's ear so that none of the persons present could hear. Thereupon I did likewise. Hans was asked to add the two. Since each of the experimenters knew only his own number, the sum, if known to anyone, could be known to Hans alone. Every such test was immediately repeated with the result known to the experimenters. In 31 tests in which the method was procedure without knowledge, 3 of the horse's answers were correct, whereas in the 31 tests in which the method was procedure with knowledge, 29 of his responses were correct. Since the three correct answers in the cases in which procedure was without knowledge evidently were accidental, the results of this series of experiments show that Hans was unable to solve arithmetical problems.

   • Reading a design for validity: Source 1, paragraph 1

     Why did Pfungst alternate tests in which the questioner knew the answer with tests in which no one did?

     Result: Alternating spreads fatigue, mood and setting across both conditions, so a difference between them can be traced to the questioner's knowledge. Pfungst notes that earlier single trials "could not claim objective validity, even though the persons making the tests were subjectively convinced": confidence is not evidence.

   • From percentages back to counts: Source 1, paragraph 3

     Pfungst reports 8% correct in 49 tests without knowledge and 98% in 42 tests with knowledge. How many correct answers is that, and what was the test really measuring?

     Result: 49 x 0.08 = 3.92, so 4 correct (4/49 = 8.2%; 3 or 5 would round to 6% or 10%). 42 x 0.98 = 41.16, so 41 correct (41/42 = 97.6%). A "reading test" that tracks whether the questioner knew the answer is measuring the questioner, not the horse: a construct validity failure.

   • Systematic error and a comparison (the warm-up thermometer)

     The thermometer reads 70 degrees when the room is really 68, and 76 when it is really 74. Which uses of the data are still valid?

     Result: Each reading is 2 degrees too high, so a monthly mean is also 2 degrees too high. The change is 76 - 70 = 6 degrees, the same as the true change 74 - 68 = 6, because a constant bias cancels in a difference. Reliable, biased for absolute values, still usable for changes.

   • Random error and a group difference

     A reaction-time app (invented example) records each trial up to 50 milliseconds too early or too late, at random. Two groups of 5 students truly differ by 20 milliseconds on average. What can go wrong?

     Result: Each recorded trial can land anywhere in a 100-millisecond window around the true time, a window 5 times as wide as the real 20-millisecond difference. With only 5 students per group, the noise can hide or even reverse the difference. Fixes: a more precise timer, or many trials per student averaged together.

   • Naming the kind of validity at risk

     Name the problem: (a) a "creativity score" that counts how many words a student writes in 2 minutes; (b) a study of 20 college athletes reported as true for "all adults"; (c) a memory study in which the music group also got 5 extra minutes of study time.

     Result: (a) Construct validity: word count is speed, not creativity. (b) External validity: the sample is too narrow to generalize. (c) Internal validity: the extra time is a confound that could explain any difference.

   Close the segment with Diagram 1 and ask students to place the thermometer, the reaction-time app and the Hans "reading test" on one of its four targets.

3. Guided Practice15 minutes

   Read Source 4 together. It is an invented study with invented data, and a small sample: 20 pedestrians per round at one corner, so its rates describe those people only and are not estimates for any city. The observers recorded public behavior without names or photos and without interacting with anyone; even so, a real study like this would need approval from a research review board.

   1Invented study summary, written for this page. A research team wanted to know how often people waiting at one downtown crosswalk look at a phone while they wait. Two trained observers stood at the same corner at the same time and coded the same 20 pedestrians, without talking to each other. No names, photos or other identifying details were recorded, and the observers did not interact with anyone.

   2Round 1 used this definition: "A person counts as using a phone if they seem distracted by it." Observer A coded 12 of the 20 pedestrians as using a phone, and Observer B coded 9. The two observers gave the same code for 17 of the 20 pedestrians: both said "phone" for 9 people and both said "no phone" for 8. For the other 3 people, A said "phone" and B said "no phone."

   3The team then rewrote the definition: "A person counts as using a phone if their eyes are on a screen for at least 2 seconds while they wait for the signal." Round 2 took place a week later, with 20 new pedestrians. Observer A coded 11 as using a phone and Observer B coded 10. The observers agreed on 19 of the 20 pedestrians: both said "phone" for 10 and both said "no phone" for 9. For the one remaining person, A said "phone" and B said "no phone."

   4The team's note: "Before we report a phone-use rate, we need to know that the rate does not depend on which of us was watching."

   1. Agreement in round 2. Percent agreement is the number of cases both observers coded the same, divided by the number of cases. Round 2: 10 + 9 = 19 agreements out of 20, so 19 / 20 = 95%. Have pairs work out round 1 the same way on their own; it returns in the quiz.
   2. The rate depends on who watched. In round 1, Observer A would report a phone-use rate of 12 / 20 = 60% and Observer B 9 / 20 = 45%. The 15-point gap comes from the observers alone, because they watched the same 20 people at the same moment. Any analysis built on one observer's numbers would carry that gap with it.
   3. Check the counts. In round 1, A's 12 "phone" codes are the 9 shared ones plus the 3 disagreements, and B's 9 are the shared 9: the paragraph's numbers fit together. Ask pairs to run the same check on round 2.
   4. Agreement is not accuracy. Ask: if both observers shared the same blind spot, for example missing phones held low, would their agreement reveal it? (No. High agreement shows reliability; it cannot show that the codes match what people were really doing.) Keep this question open for Source 2.
4. Independent Practice15 minutes

   Students read Sources 2, 3 and 5 on their own and use them, with Sources 1 and 4, for the quiz and homework. Source 2 is the report a group of 13 observers signed in September 1904, before Pfungst's tests. Source 3 is Pfungst's account of what he found. Source 5 is an invented study with invented data from only 8 volunteers, a small sample: its numbers can show how to reason about reliability and validity, but they could not settle the question for any real questionnaire. Pfungst's tests, too, involved one horse and a few questioners, and his laboratory check used 25 people.

   1"The undersigned came together for the purpose of investigating the question whether or not there is involved in the feats of the horse of Mr. von Osten anything of the nature of tricks, that is, intentional influence or aid, on the part of the questioner. After a careful investigation they are unanimously agreed that such signs are out of the question under the conditions which were maintained during this investigation. This decision in no wise takes into account the character of the men exhibiting the horse, and who are known to most of the undersigned: In spite of the most attentive observation, nothing in the way of movements or other forms of expression which might have served as a sign, could be discovered. In order to obviate involuntary movements on the part of those present, one series of tests was made with only Mr. Busch present. Among these tests were some in which, according to his professional judgment, the possibility of tricks of the sort commonly used in training, was excluded. Another series of tests was made in such a way that the correct answers to the questions which Mr. von Osten put to the horse, were unknown to the questioner. From previous observation the greater number of the undersigned also know of a large number of cases in which, during the absence of Mr. von Osten and Mr. Schillings, other persons were likewise able to obtain correct responses from the horse. Among these were some cases in which the questioner did not know the correct solution of the problem or was mistaken about it. And lastly, several of the undersigned have become acquainted with the method which Mr. von Osten used, which has little in common with methods of training, and is patterned after the instruction given in the elementary schools. As a result of these observations the undersigned are of the opinion that unintentional signs of the kind which are at present familiar, are likewise excluded. They are unanimously agreed that this much is certain: This is a case which appears in principle to differ from any hitherto discovered, and has nothing in common with training, in the usual sense of that word, and therefore is worthy of a serious and incisive investigation.

   1Investigations of the other senses became needless, for I had, in the meantime, succeeded in discovering the essential and effective signs in the course of my observations of Mr. von Osten. These signs are minimal movements of the head on the part of the experimenter. As soon as the experimenter had given a problem to the horse, he, involuntarily, bent his head and trunk slightly forward and the horse would then put the right foot forward and begin to tap, without, however, returning it each time to its original position. As soon as the desired number of taps was given, the questioner would make a slight upward jerk of the head. Thereupon the horse would immediately swing his foot in a wide circle, bringing it back to its original position. (This movement, which in the following exposition we shall designate as "the back step", was never included in the count.) Now after Hans had ceased tapping, the questioner would raise his head and trunk to their normal position. This second, far coarser movement was not the signal for the back-step, but always followed it. But whenever this second movement was omitted, Hans, who had already brought back his foot to the original position and had thereby put it out of commission, as it were, would give one more tap with his left foot.

   2If it was true that these movements of the questioner guided the horse in his tapping, then the following must be shown: First, that the same movements were observed in Mr. von Osten in every case of successful response; secondly, that they recurred in the same order or with only slight individual changes in the case of all who were able to obtain successful responses from the horse, and that they were absent or occurred at the wrong time in all cases of unsuccessful response. Furthermore, it was observed that it was possible to bring about unsuccessful reactions on the part of the horse as soon as the movements were voluntarily suppressed, and conversely, that by voluntarily giving the necessary signs the horse might be made to respond at pleasure; so that anyone who possessed the knowledge of the proper signs could thereby gain control over the process of response on the part of the horse. These requirements have all been fulfilled, as we shall see in the following pages.

   [...]

   3[...] I thus tested, all in all, twenty-five persons, of every age and sex (including children of five and six years), differing also in nationality and occupation. None of them was aware of the purpose of the experiments. It could not escape them, to be sure, that they were being watched. It was also evident to them that the things noted were certain tensions and movements; but none of my subjects discovered what the particular phenomena were that I was looking for. Only in a few isolated instances did they report that they were conscious of any movements on their part. With the exception of two persons, they all made the same involuntary movements which were described in chapter II, the most important of which was the sudden slight upward jerk of the head when the final number was reached. [...]

   1Invented study summary, written for this page. A group of teachers wrote a 10-item "Study Focus" questionnaire. Students rate statements such as "I can keep my mind on my work for a long time" from 1 (not like me) to 5 (very like me), so totals run from 10 to 50. Eight student volunteers took part, with their own and a parent's written permission. They could skip any item or stop at any time, and their scores were kept under code letters A to H.

   2Step 1, consistency. Each volunteer took the questionnaire twice, two weeks apart. First totals, A to H: 42, 35, 47, 30, 38, 44, 33, 40. Second totals, A to H: 40, 37, 45, 33, 37, 42, 35, 43.

   3Step 2, accuracy. During one 30-minute study period, an observer who had not seen any questionnaire scores recorded how many minutes each volunteer spent on the assigned task. Minutes on task, A to H: 17, 24, 18, 19, 20, 22, 15, 25.

   4The teachers' first draft conclusion: "Our questionnaire gives almost the same score twice, so it is an accurate measure of how focused students are when they study."

   Task: in one paragraph, decide whether the 1904 observers' conclusion in Source 2 was a failure of reliability, of validity, or of both. Quote Source 2 once and Source 3 once, with paragraph numbers.

5. Closure5-10 minutes

   Exit ticket: "Give one example of a measure that is reliable but not valid, and say what it would do to an average computed from it."

   Teacher note on research ethics: Pfungst writes in Source 3, paragraph 3 that none of the 25 people he tested "was aware of the purpose of the experiments." Point out that this was ordinary practice in 1907. Today, the federal rules for research with people (the Common Rule, 45 CFR 46) require informed consent, and a study may leave out information about its purpose only when a review board approves; the rules also call for telling participants more afterward when appropriate. Students in this lesson plan studies on paper only.

   Teacher note on the sources: Sources 1-3 are from the 1911 English translation of Pfungst's book. The 1904 report rules out tricks and describes von Osten's school-style teaching method. Pfungst's explanation does not accuse anyone of trickery: Source 3, paragraph 1 says the questioner made the movements "involuntarily."

Differentiation Strategies

Assessment Guidance

1. Question 1 of 20 · Multiple Choice

   Two judges independently score the same 12 skateboard routines and give almost identical scores. This is evidence that the scoring is

   Avalid, because both judges are experts in the sport Breliable: the score barely depends on the judge Cvalid, because agreement proves the scores are right Dexternally valid, since it applies to all skaters

2. Question 2 of 20 · Multiple Choice

   In Source 1, paragraph 2, what does Pfungst mean by "procedure without knowledge"?

   AThe horse had never been trained on that kind of problem BThe questioner knew the answer but tried to hide it CNobody present, even the questioner, knew the answer DThe observers were not told the purpose of the study

3. Question 3 of 20 · Multiple Choice

   According to Source 1, paragraph 1, why could the negative results from earlier single trials "not claim objective validity"?

   AThe people who ran them had no scientific training at all BThe results of those trials were never written down CThe horse was tired during every one of those trials DThey lacked enough tests and alternating conditions

4. Question 4 of 20 · Multiple Choice

   Pfungst writes in Source 1, paragraph 4 that the few correct answers without knowledge "must be regarded as fortuitous." What does he mean?

   AThey came from signals the questioner gave too late BThey show the horse could read some numerals after all CThey were errors in the record and should be deleted DThey were lucky hits of the kind chance produces

5. Question 5 of 20 · Multiple Choice

   In Source 2 the 13 observers "are unanimously agreed" that no tricks were involved, and they rule out the familiar kinds of unintentional sign, yet Pfungst later found signs. What does this case show about observation?

   AObservers with many different jobs cannot agree on anything BObservers can all agree and still all miss the same cue CUnanimous agreement among experts proves that a finding is valid DA commission's report is less reliable than one person's notes

6. Question 6 of 20 · Multiple Choice

   In Source 3, paragraph 1, what signal told the horse to stop tapping?

   AA slight upward jerk of the questioner's head BA word whispered by the questioner into his ear CThe questioner raising the right hand toward him DThe reward of bread or carrot held out to him

7. Question 7 of 20 · Multiple Choice

   Source 3, paragraph 2 says the same movements should be "absent or occurred at the wrong time in all cases of unsuccessful response." Why does checking the failures matter?

   AFailures are easier to record than the horse's successes BFailures showed that the horse was tired or bored CFailures would let Pfungst drop the tests that went badly DA cue also seen before wrong answers explains nothing

8. Question 8 of 20 · Multiple Choice

   Source 3, paragraph 3 says none of the 25 people Pfungst tested "was aware of the purpose of the experiments." How did this protect the validity of the test?

   APeople who knew might hide their movements BIt let Pfungst finish all 25 tests in a single day CIt meant that the 25 people did not need to be paid DIt made the people more likely to agree with Pfungst

9. Question 9 of 20 · Multiple Choice

   Use Source 4, paragraph 2. What was the observers' percent agreement in round 1?

   A45% B60% C15% D85%

10. Question 10 of 20 · Multiple Choice

    In Source 4, why did agreement rise from round 1 to round 2?

    ARound 2 had more pedestrians, so the estimate was steadier BThe observers compared notes while they were coding CThe new definition named an observable act and a time DThe observers coded different people in round 2

11. Question 11 of 20 · Multiple Choice

    Source 5, paragraph 2 describes giving the same questionnaire to the same volunteers two weeks apart. Which property does this step check?

    AInter-rater reliability BTest-retest reliability CCriterion validity DExternal validity

12. Question 12 of 20 · Multiple Choice

    In Source 5, paragraph 3, the observer "had not seen any questionnaire scores." Why does that matter?

    AIt kept expectations from shaping the minutes recorded BIt made the observer's job faster during the study period CIt meant the questionnaire no longer needed to be scored DIt ensured that every student would be on task

13. Question 13 of 20 · Multiple Choice

    A step-counting app adds about 500 extra steps to every day's total. A student compares her mean daily steps in September with October. Which statement is accurate?

    ABoth monthly means and the difference between them are wrong BThe app is unreliable, so neither month's data can be used CEach mean is too high, but the difference is accurate DThe extra steps cancel out once a full month is averaged

14. Question 14 of 20 · Multiple Choice

    A sleep diary has a lot of random error: students often guess their bedtime. A researcher correlates diary sleep hours with grades. Compared with the true relationship, the correlation found will most likely be

    Astronger, because guessing adds more variation to the data Babout the same, since random errors always cancel out Cweaker, because noise hides part of the real pattern Dreversed in sign, because guesses are usually too high

15. Question 15 of 20 · Short Answer

    Use Source 3, paragraph 3. What percent of the 25 people Pfungst tested made the same involuntary movement? Explain what testing people "of every age and sex," of different nationality and occupation, adds to his explanation.

    Show model answer

16. Question 16 of 20 · Short Answer

    Use Source 4, paragraph 3. What phone-use rate would each observer report for round 2? Explain why the team wanted high agreement before reporting any rate.

    Show model answer

17. Question 17 of 20 · Short Answer

    Use Source 5, paragraph 2. Compute the mean of the first totals and the mean of the second totals. Explain why two similar means are not, by themselves, enough to show test-retest reliability.

    Show model answer

18. Question 18 of 20 · Short Answer

    Explain the difference between random error and systematic error. For each, give an example from this lesson and say whether collecting more data would reduce it.

    Show model answer

19. Question 19 of 20 · Short Answer

    Pfungst's "procedure without knowledge" is an early form of a blind procedure. Explain the idea, and describe, on paper only, how an invented study of a new reading app could use a blind observer.

    Show model answer

20. Question 20 of 20 · Short Answer

    Explain why a measure can be reliable without being valid, but cannot be highly valid unless it is reliable. Use Source 2 as one example.

    Show model answer

These standards connect to D2.Psy.5.9-12: prerequisites to review first, parallel standards at the same level, and next steps that build on it.

Standard text: National Council for the Social Studies (NCSS), The College, Career, and Civic Life (C3) Framework for Social Studies State Standards: Guidance for Enhancing the Rigor of K-12 Civics, Economics, Geography, and History (Silver Spring, MD: NCSS, 2013). NCSS was not involved in the production of this lesson and does not endorse it.