D2.Soc.5.9-12: Strengths and Weaknesses of Surveys, Experiments, Observation and Content Analysis

Overview

Lesson Procedure

60-70 minutes of class time across 5 phases.

1. Warm-Up5-10 minutes
2. Direct Instruction20 minutes
3. Guided Practice15 minutes
4. Independent Practice15 minutes
5. Closure5-10 minutes

1. Warm-Up5-10 minutes

   Warm-Up Prompt

   "You want to know how much time students at your school spend on homework each night. Name two different ways to find out. What could make each way give a wrong answer?"

   List the ideas on the board and group them: asking students (a survey), watching or keeping diaries (observation), changing something and comparing (an experiment), and counting what already exists, such as assignment calendars (content analysis). Each idea has a way of going wrong; today's question is what those ways are for each method.

2. Direct Instruction20 minutes

   Four methods of sociological research (summary written for this page)

   Method	What the researcher does	A typical strength	A typical weakness
   Survey	Asks a sample of people the same questions, by questionnaire or interview	Reaches many people at modest cost; a random sample can stand for a larger population	People may misremember, shade the truth or not answer at all
   Experiment	Changes one condition for a randomly assigned group and compares it with a control group	Gives the strongest evidence that one thing causes another	The setting can be artificial, and many causes cannot ethically be assigned
   Observation	Watches and records behavior where it happens, sometimes while taking part	Shows what people actually do, in depth and in context	Groups are small and not randomly chosen; the observer can change what happens
   Content analysis	Sorts and counts what texts, images or media contain, using fixed categories	The material already exists, so studying it disturbs no one, and others can recount it	Shows what was produced, not how audiences understood it; categories shape the result

   Key terms. A population is the whole group a researcher wants to learn about; a sample is the part actually studied. In a random sample, every member of the population has a known chance of being chosen. The response rate is the share of people chosen who actually take part. In an experiment, random assignment puts participants into groups by chance, the independent variable is the condition the researcher changes, the dependent variable is the outcome measured, and the control group does not get the change. In content analysis, coding means placing each item in a category, and intercoder agreement is how often two coders working separately place items in the same category.

   Read Source 1. The Youth Risk Behavior Survey (YRBS) is a CDC survey of high school students; Source 1 comes from the CDC's 2013 report on how the survey is run. The survey asks about health behaviors, but this lesson uses only the parts that explain its method.

   1In 2000, CDC conducted a study to assess the validity of the two YRBS questions on self-reported height and weight (25). In that study, 2,965 high school students completed the 1999 version of the YRBSS questionnaire on two occasions approximately 2 weeks apart. After completing the questionnaire, the students were weighed and had their height measured. Self-reported height, weight, and BMI calculated from these values were substantially reliable, but on average, students in the study overreported their height by 2.7 inches and underreported their weight by 3.5 pounds, which indicates that YRBSS probably underestimates the prevalence of overweight and obesity in adolescent populations.

   [...]

   2The final stage of sampling consists of randomly selecting one or two entire classes in each chosen school and in each of grades 9–12. Examples of classes include homerooms or classes of a required subject (e.g., English and social studies). All students in sampled classes are eligible to participate. Since 1991, the national YRBS has been conducted 11 times with an average sample size of 14,517 and average school, student, and overall response rates of 78%, 86%, and 71%, respectively (Table 3).

   [...]

   3Procedures for all the YRBSs are designed to protect student privacy by allowing for anonymous and voluntary participation. In all surveys, students complete the self-administered questionnaire during one class period and record their responses directly in a computer-scannable booklet or on a computer-scannable answer sheet. To the extent possible, students' desks are spread throughout the classroom to minimize the chance that students can see each other's responses. Students also are encouraged to use an extra sheet of paper or an envelope provided by the data collector to cover their responses as they complete the questionnaire. In the national survey, and in certain state, territorial, tribal, and large urban school district surveys, when students complete the questionnaire, they are asked to seal their questionnaire booklet or answer sheet in an envelope before placing it in a box.

   [...]

   4YRBSS is subject to at least five limitations. First, all YRBSS data are self-reported, and the extent of underreporting or overreporting of behaviors cannot be determined, although studies described in this report demonstrate that the data are of acceptable quality. Second, the school-based national, state, territorial, tribal, and large urban school district survey data apply only to youths who attend school and therefore are not representative of all persons in this age group. Nationwide, in 2009, approximately 4% of persons aged 16–17 years were not enrolled in a high school program and had not completed high school (88). The NHIS and Youth Risk Behavior Supplement conducted in 1992 demonstrated that out-of-school youths are more likely than youths attending school to engage in the majority of health-risk behaviors (89). [...]

   Then read Source 3. It is the preface to Nels Anderson's book The Hobo (1923), written by the Chicago committee that sponsored his study, and one numbered field note from the book. "Hobo" was the word of the time for a migrant worker who traveled from job to job, often by freight train; "Hobohemia" was the name for the city districts where such men lived. Both are dated terms. A life-history is a long personal account a researcher collects from one person. Sample warning: the study rests on about sixty life-histories gathered informally in one city, so it cannot give percentages for migrant workers as a whole.

   1The Committee on Homeless Men was organized by the Executive Committee of the Chicago Council of Social Agencies on June 16, 1922, to study the problem of the migratory casual worker. Its members included men and women in contact with the problem of homeless men from different points of view.

   2Mr. Nels Anderson, a graduate student in sociology in the University of Chicago, was selected to make the study. Mr. Anderson was already thoroughly familiar with the life of the migratory casual worker. He had shared their experiences “on the road” and at work, and had visited the Hobohemian areas of many of the large western cities. In the summer of 1921, he made a study of 400 migrants. Early in 1922, through the generous assistance and encouragement of Dr. William A. Evans, Dr. Ben L. Reitman, and Joel D. Hunter, he began a study of homeless men in Chicago, in connection with a field-study course at the University of Chicago.

   [...]

   3The object of this inquiry, from the standpoint of the Committee, was to secure those facts which would enable social agencies to deal intelligently with the problems created by the continuous ebb and flow, out of and into Chicago, of tens of thousands of foot-loose and homeless men. Only through an understanding both of the human nature of the migratory casual worker, and of the economic and social forces which have shaped his personality, could there be devised any fundamental program for social agencies interested in his welfare.

   4Earlier studies of the migratory casual workers in the United States have been limited almost entirely to statistical investigation. In the present inquiry a more intensive study of cases was decided upon in preference to an extensive statistical survey. For the past twelve months Mr. Anderson lived in Hobohemia, and in a natural and informal way secured upward of sixty life-histories, and collected, in addition, a mass of documents and other materials which are listed in Appendix B. Mr. Anderson has had, in certain parts of the field work, the assistance of [...] students in sociology at the University of Chicago, and has utilized the results of past studies of this subject by students in the department.

   [...]

   57. An investigator became acquainted with two men who were jack rollers who operated on Madison Street west of Halsted. They were well dressed for the “street” though not so well groomed as to be conspicuous. The investigator pretended to them that he had just spent ninety days in the jail in Salt Lake City for “rolling” a drunk. [...]

   • A survey's strength and weakness: Source 1, paragraph 2

     What do the averages at the end of paragraph 2 tell you about how well the national YRBS sample works?

     Result: Averaging 14,517 students, the sample is large and drawn by chance, a strength: results can stand for high school students nationwide. The average overall response rate was 71%, so about 100 - 71 = 29% of the students sampled were not counted, a weakness if they differ from those who took part.

   • Observation's strength and weakness: Source 3, paragraph 4

     What did the Chicago study choose instead of a statistical survey, and what did it gain and lose?

     Result: It chose "a more intensive study of cases": Anderson lived in Hobohemia for twelve months and "secured upward of sixty life-histories." The strength is depth: how the men lived and why, told in their own words in their own setting. The weakness is reach: sixty informally chosen men in one city cannot show how common any pattern is among all migrant workers.

   • An experiment (invented example)

     Invented: a researcher randomly assigns 60 volunteers to two equal groups. One reads a news story with a photo, the other the same story without it; then each rates how much they trust the story. Why assign by chance, and what is a weakness?

     Result: Each group has 60 / 2 = 30 people. Random assignment makes the groups alike on average in age, interests and habits, so a difference in trust can be credited to the photo, the independent variable. A weakness: reading one story in a study room, as a volunteer, is not how people meet news at home, so the result may not carry over.

   • Content analysis (invented example)

     Invented: a student codes 40 cereal-box covers and finds a cartoon character on 26 of them. What share is that, and what can the count not tell?

     Result: 26 / 40 = 0.65, or 65%. A strength: the boxes already exist, and anyone can recount them with the same rule. A weakness: the count describes what the boxes show, not whether the characters change what children ask their families to buy.

   • Matching a method to a question

     Do students sit with the same people at lunch every day? Which method fits best, and why not the others?

     Result: Structured observation over several days (recording seating, not names) shows what students actually do. A survey would be quicker but depends on memory and on how students want to be seen. An experiment that assigned seats would change the very behavior being studied.

   Close the segment: Which method would you trust most to show that something causes something else, and which to show what people's lives are like? Ask students to name the trade-off, not a single winner.

3. Guided Practice15 minutes

   Read Source 5 together. It is an invented experiment with invented data, set in an invented school.

   1Invented study summary, written for this page. A researcher asked whether a poster describing what other students do changes how much paper students recycle. In an invented high school, the names of 16 homerooms were drawn from a hat. The first 8 drawn got a poster reading "Most students at this school recycle their paper" (the message poster). The other 8 got a poster showing only the recycling symbol and the words "Recycle paper here" (the plain poster). Every homeroom had about 25 students and the same kind of bin.

   2For four weeks, custodians weighed each room's recycling bin every Friday. The custodians were not told which poster each room had. The invented results below are each homeroom's average weekly weight of recycled paper, in kilograms.

   3Message poster: 2.4, 3.1, 2.8, 2.2, 3.5, 2.9, 2.6, 3.0.

   4Plain poster: 2.1, 2.5, 1.9, 2.7, 2.3, 2.0, 2.8, 2.2.

   5The principal approved the study, and students were told about it when it ended. Researcher's note: this was one school over four weeks, and the teachers knew which poster their room had.

   1. Name the parts. What is the independent variable, what is the dependent variable, and which group plays the role of the control group?
   2. Random assignment. What did drawing names from a hat do that letting teachers choose a poster would not?
   3. Blind measurement. Why does it matter that the custodians were not told which poster each room had?
   4. Look at the data. Diagram 1 shows every homeroom. Without computing, which group's results look higher, and do the two groups overlap?
   5. Judge it. Use the researcher's note to name two reasons to be careful before saying the message poster would work in other schools. (Students compute the two means in the homework.)
4. Independent Practice15 minutes

   Students read Sources 2, 4 and 6 on their own and use all six sources for the quiz and homework. Source 2 is a real experiment reported by the CDC, Source 4 a 1926 account of content analysis, and Source 6 an invented content analysis. Sample warning: Source 6 covers one invented paper, and Source 2's excerpt gives no sample sizes, so neither supports claims about how large an effect is in general.

   1In 2004, CDC conducted a study to determine how varying the mode and setting of survey administration might affect prevalence estimates. While previous research had examined the effects of varying setting (school versus home) (76–79) and mode (paper-and-pencil instrument [PAPI] versus computer-assisted self-interview [CASI]) (80–83), this study was the first to assign school classes randomly to one of four conditions in which mode and setting were varied systematically: school-based administration using PAPI, school-based administration using CASI, home-based PAPI administration, and home-based CASI administration. Results revealed that students completing questionnaires at school were more likely to report health-risk behaviors than students completing questionnaires at home, but that mode effects were weaker: prevalence estimates for health-risk behaviors generally did not differ for CASI and PAPI administrations, and these effects were independent of setting (84). On the basis of these results, CDC decided to continue with PAPI administration for the YRBSS. Because the use of CASI did not increase the reporting of risk behaviors, its increased cost and complicated logistics did not appear to be justified.

   2In 2008, CDC conducted two additional studies to determine the feasibility and effect of conducting YRBS as a web-based survey. In the first study, classes in grades 9 and 10 were assigned randomly to complete the YRBS in three conditions: 1) using PAPI in the classroom, 2) using web-based CASI administration in school computer labs or in classrooms with sufficient numbers of computers, or 3) using web-based CASI that students could complete on their own (i.e., at any time at any computer with Internet access). Results indicated that risk behavior prevalence estimates generated from PAPI and CASI administered in a classroom setting in schools generally were equivalent (85). However, web-based CASI administration yielded more missing data than PAPI administration, and web-based "on their own" administration yielded an unacceptably low response rate. [...]

   1It is now necessary to describe the routine employed in classifying the news content of the Connecticut weeklies, 35 in number.

   2The unit of measurement. First a unit of measurement had to be established. Johnson, as well as others who had analyzed the daily papers, found that a count of items was all that is needed. They ascertained the number of items in each category. Because of the “law of averages” operating in the large number of items included, this gave the same result as an actual measurement of column inches. The item was defined by Johnson as “any amount of news space beginning with a special head in heavy faced type and ending with lead, column lines, an asterisk or the end of a column not continued, provided that the next column begins with a new head.”

   3While this method of classification can be used with the daily newspaper, it cannot be used with the weekly newspaper, where news material is often printed without headlines, and where large amounts of “personal” matter—short items relating to individuals, their visits, their guests, etc.—would give an undue weight to this class were the item method used. All balance would be destroyed were each personal item of this type counted as a unit. A preliminary study made this clear. A single column of news may contain in a weekly newspaper as many as 30 “personals.” Obviously this would give excessive importance to this class, and it would not be justifiable to balance these 30 items against a single column item concerning a town meeting.

   4It was essential to measure all material in the country paper on the space basis. The actual number of column inches of printed matter to the nearest half inch was recorded for each issue of the paper studied. Advertising was not included; in classifying, only news and magazine material were considered.

   [...]

   5The period covered by the study. Having determined the unit of measurement (column inches rather than items), the period to be covered by the analysis had to be established. Taylor’s study limited the analysis to a single week for a large number of papers. This makes any study of seasonal variation impossible. Neither does it preclude the possibility that a single issue of a weekly paper is not a balanced sample of the paper. It is highly improbable that one issue is typical of the make-up of the country paper.

   1Invented study summary, written for this page. Two students used Willey's space method (Source 4) on an invented weekly paper, the Riverton Ledger. They chose one issue from each month of one year, 12 issues in all, and measured every news item, headline included, in column inches. Advertising was left out.

   2They sorted the space into six categories. The invented totals for the 12 issues were: local events, 384 column inches; personals (short items about residents' visits, guests and trips), 216; sports, 240; school news, 144; town government, 120; and business, 96. The total was 1,200 column inches.

   3To test their categories, both students first coded the same 50 items separately. They agreed on 46 of them. Most of the disagreements were items that fit two categories, such as a story about the town paying for new library computers.

   4Researchers' note: the study shows what the paper printed, not what its readers read, believed or did.

   Task: make a four-row chart with the headings Method, Source, One strength the source shows, and One weakness the source shows. Fill one row each for a survey, an experiment, an observation study and a content analysis, quoting a few words from the source in each box.

5. Closure5-10 minutes

   Exit ticket: "Pick one of the four methods. Write one strength and one weakness, each with a piece of evidence from today's sources."

   Teacher notes. (1) Source 3 uses period terms: "hobo," "Hobohemia," "migratory casual worker" and, in the field note, "jack rollers," 1920s slang for men who robbed drunk or sleeping people. The rest of that field note, which quotes the two men, is left out. Present these as historical vocabulary. (2) The field note describes an investigator who lied about himself to gain trust. Today, research with people is reviewed in advance, usually by an institutional review board, and deception is allowed only under strict conditions, with a debriefing afterward; use the note as a case to evaluate, not a model. (3) The YRBS asks about sensitive health behaviors; this lesson uses only its method sections, and no student is asked about personal behavior. (4) Sources 5 and 6 are invented, and every study students design is planned on paper only, with no data collected from classmates.

Differentiation Strategies

Assessment Guidance

1. Question 1 of 20 · Multiple Choice

   According to Source 1, paragraph 2, how are students chosen within each sampled school?

   ATeachers pick a few students from every grade BStudents sign up online if they wish to join COne or two whole classes are chosen at random DOnly students with high attendance are invited

2. Question 2 of 20 · Multiple Choice

   In Source 1, paragraph 3, why are students' desks spread throughout the classroom?

   ATo keep students from seeing each other's answers BTo let the data collector watch each student work CTo fit the whole survey into one class period DTo stop students from talking about the questions

3. Question 3 of 20 · Multiple Choice

   According to Source 1, paragraph 4, which young people does the school-based survey leave out?

   AStudents in the ninth grade BYouths not enrolled in school CStudents who live in big cities DStudents who answer on paper

4. Question 4 of 20 · Multiple Choice

   What does the height-and-weight check in Source 1, paragraph 1 show about those self-reported answers?

   AThey changed so much between the two times that they were useless BThey were exactly right once the students were weighed and measured CThey were too few to say anything about high school students DThey were consistent from time to time, yet tilted in one direction

5. Question 5 of 20 · Multiple Choice

   In Source 2, paragraph 1, what made the 2004 study an experiment rather than a comparison of existing surveys?

   AStudents chose to answer at home or at school BSchool classes were randomly assigned to conditions CResults of earlier surveys were compared over time DEvery student answered once on paper and once online

6. Question 6 of 20 · Multiple Choice

   What did the 2004 study in Source 2, paragraph 1 find about where students took the survey?

   AStudents reported more risk behaviors at school than at home BStudents reported more risk behaviors at home than at school CStudents reported more risk behaviors on computers than on paper DStudents reported the same risk behaviors in every condition

7. Question 7 of 20 · Multiple Choice

   Which weakness of letting students answer the web survey "on their own" does Source 2, paragraph 2 report?

   AAnswers were more honest than in class BThe cost was higher than for paper forms CToo few students chose to take part DResults matched the paper results exactly

8. Question 8 of 20 · Multiple Choice

   According to Source 3, paragraph 3, what did the committee want the study to help social agencies do?

   ACount every homeless man living in Chicago BDeal intelligently with the men's problems CPersuade the men to settle in a single city DFind steady factory jobs for all of the men

9. Question 9 of 20 · Multiple Choice

   Which detail in Source 3, paragraph 2 points to a strength of Anderson as an observer?

   AHe had never met a migrant worker before BHe worked only from records the police kept CHe was a doctor in the city health department DHe had shared their life on the road

10. Question 10 of 20 · Multiple Choice

    According to Source 3, paragraph 1, what was true of the committee's members?

    AThey saw the problem from different viewpoints BThey had all lived as migrant workers themselves CThey were all professors at the same university DThey had never before met a homeless man

11. Question 11 of 20 · Multiple Choice

    In Source 4, paragraph 3, why could Willey not simply count items in weekly papers?

    AWeekly papers printed too few items to count BShort personals would get far too much weight CWeekly papers mixed advertising into stories DOld weekly papers had not been saved to study

12. Question 12 of 20 · Multiple Choice

    What weakness of Taylor's single-week study does Willey name in Source 4, paragraph 5?

    AIt counted the headlines as part of every item BIt covered too few newspapers to compare them CIt could not show changes from season to season DIt measured space instead of counting the items

13. Question 13 of 20 · Multiple Choice

    In Source 6, what percent of all the column inches went to sports?

    A18% B24% C32% D20%

14. Question 14 of 20 · Multiple Choice

    In Source 5, how many message-poster homerooms collected more than the highest plain-poster homeroom?

    A3 B5 C8 D4

15. Question 15 of 20 · Short Answer

    Using two different paragraphs of Source 1, explain one strength and one weakness of the YRBS as a survey. Quote or cite a number for each.

    Show model answer

16. Question 16 of 20 · Short Answer

    Sources 2 and 5 both describe experiments. For each, explain which feature of the design lets the researchers credit a difference to one cause, and give one weakness.

    Show model answer

17. Question 17 of 20 · Short Answer

    Using Source 3, explain one strength and one weakness of the participant observation Anderson used, and one way an observer's closeness to the people studied could shape what gets recorded.

    Show model answer

18. Question 18 of 20 · Short Answer

    Explain how Willey's choices of unit and study period in Source 4 deal with weaknesses of content analysis, whether Source 6 followed those choices, and one weakness that remains in Source 6.

    Show model answer

19. Question 19 of 20 · Short Answer

    Invented question: a town wants to know whether its new teen center reduces loneliness among teenagers. Choose a method and plan the study on paper only. Give one strength and one weakness of your method and of one method you rejected.

    Show model answer

20. Question 20 of 20 · Short Answer

    Invented case: a school posts an optional online form about the cafeteria. Of its 800 students, 40 answer, and 36 of them say they like the cafeteria. Find the response rate and the share of all 800 students who said they like it. Then explain two weaknesses of this survey and one change, modeled on Source 1, that would fix one of them.

    Show model answer

These standards connect to D2.Soc.5.9-12: prerequisites to review first, parallel standards at the same level, and next steps that build on it.

Standard text: National Council for the Social Studies (NCSS), The College, Career, and Civic Life (C3) Framework for Social Studies State Standards: Guidance for Enhancing the Rigor of K-12 Civics, Economics, Geography, and History (Silver Spring, MD: NCSS, 2013). NCSS was not involved in the production of this lesson and does not endorse it.