Actions

Difference between revisions of "Survey Research"

From OPOSSEM

Line 32: Line 32:
 
=== Example of Framing  ===
 
=== Example of Framing  ===
  
In one poll of adults 18 and older in York County, SC conducted in 2003, two versions of one question were included to demonstrate the phenomenon of framing. Respondents were randomly selected to receive one of the versions. In a series of questions where respondents were asked to Strongly Agree, Agree, Neither Agree nor Disagree, Disagree, or Strongly Disagree with a statement, half of the respondents received Version A of one statement while half received Version B.  
+
In one telephone poll of adults 18 and older in York County, SC conducted in 2003, two versions of one question were included to demonstrate the phenomenon of framing. Respondents were randomly selected to receive one of the versions. In a series of questions where respondents were asked to Strongly Agree, Agree, Neither Agree nor Disagree, Disagree, or Strongly Disagree with a statement, half of the respondents received Version A of one statement while half received Version B.  
  
 
<u>Version A</u>: "'''I don't mind if the government keeps tabs on regular people'''."  
 
<u>Version A</u>: "'''I don't mind if the government keeps tabs on regular people'''."  
Line 58: Line 58:
 
=== Example of Framing Related to Word Choice  ===
 
=== Example of Framing Related to Word Choice  ===
  
In the fall of 2008, the United States Congress was considering The Troubled Asset Relief Program (TARP), the Bush administration's plan to purchase assets and equity from financial institutions in hopes of stabilizing the financial industry. It was signed into law on October 3, 2008. In September 2008, three separate national surveys (Pew, LA Times/Bloomberg, and ABC News/Washington Post) sought to gauge public opinion regarding this program. Each survey used different language to describe the program.  
+
In the fall of 2008, the United States Congress was considering The Troubled Asset Relief Program (TARP), the Bush administration's plan to purchase assets and equity from financial institutions in hopes of stabilizing the financial industry. It was signed into law on October 3, 2008. In September 2008, three separate national telephone surveys (Pew, LA Times/Bloomberg, and ABC News/Washington Post) sought to gauge public opinion regarding this program. Each survey used different language to describe the program.  
  
 
<u>Pew wording</u>: "As you may know, the government is potentially investing billions to try and keep financial institutions and markets secure. Do you think this is the right thing or the wrong thing for the government to be doing?"  
 
<u>Pew wording</u>: "As you may know, the government is potentially investing billions to try and keep financial institutions and markets secure. Do you think this is the right thing or the wrong thing for the government to be doing?"  
Line 144: Line 144:
 
|}
 
|}
  
 +
<br>
  
 
+
In 1948, as fears surrounding the Cold War grew, individuals who were first asked about allowing a Communist reporter into the United States showed little willingness to open the country to them with only a bit more than one-third favoring admitting the Communist reporter. &nbsp;However, when asked about allowing in a Communist reporter '''after''' being asked whether Communist countries should permit the entrance of an American reporter, more than 7 out of 10 favored allowing in a Communist reporter. &nbsp;Thinking about Communist countires allowing in an American reporter generated feelings of reciprocal obligation and made respondents more likely to respond favorably to the Communist reporter item. &nbsp;Howard Schuman and Stanley Presser repeated the experiment exactly in 1980 with nearly as dramatic results.<ref>{{cite bookfckLR|last1 = SchumanfckLR|first1 = HowardfckLR|authorlink1 = fckLR|last2 = PresserfckLR|first2 = StanleyfckLR|authorlink2 =fckLR|title = Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and ContextfckLR|publisher = Sage PublicationsfckLR|series = fckLR|year = 1996fckLR|doi = fckLR|isbn = fckLR}}</ref>  
In 1948, as fears surrounding the Cold War grew, individuals who were first asked about allowing a Communist reporter into the United States showed little willingness to open the country to them with only a bit more than one-third favoring admitting the Communist reporter. &nbsp;However, when asked about allowing in a Communist reporter '''after''' being asked whether Communist countries should permit the entrance of an American reporter, more than 7 out of 10 favored allowing in a Communist reporter. &nbsp;Thinking about Communist countires allowing in an American reporter generated feelings of reciprocal obligation and made respondents more likely to respond favorably to the Communist reporter item. &nbsp;Howard Schuman and Stanley Presser repeated the experiment exactly in 1980 with nearly as dramatic results.<ref>{{cite book
 
|last1 = Schuman
 
|first1 = Howard
 
|authorlink1 =  
 
|last2 = Presser
 
|first2 = Stanley
 
|authorlink2 =
 
|title = Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context
 
|publisher = Sage Publications
 
|series =  
 
|year = 1996
 
|doi =  
 
|isbn =  
 
}}</ref>
 
  
 
<br>  
 
<br>  

Revision as of 06:27, 9 July 2011



Objectives

  • Would you say that you approve or disapprove of survey research?

Introduction

Heading

Sub-heading

Example

Sub-heading

Example

Response Effects

Response Effects refer to the phenomenon where something about the survey process impacts the responses given by survey respondents. While it is impossible to eliminate all response effects, it is possible to reduce or eliminate the structural causes of many response effects (that is, when the structure of the survey, such as the order of questions or alternatives, the wording of questions, etc. is the cause) and minimize the impact of others (for example, response effects caused by the race or sex of interviewer can be minimized by randomization in the sampling process)

Framing

Framing refers to the use of language, words, or phrases which are likely to lead survey respondents to give certain types of answers.

Example of Framing

In one telephone poll of adults 18 and older in York County, SC conducted in 2003, two versions of one question were included to demonstrate the phenomenon of framing. Respondents were randomly selected to receive one of the versions. In a series of questions where respondents were asked to Strongly Agree, Agree, Neither Agree nor Disagree, Disagree, or Strongly Disagree with a statement, half of the respondents received Version A of one statement while half received Version B.

Version A: "I don't mind if the government keeps tabs on regular people."

Results:
Error creating thumbnail: Unable to save thumbnail to destination


Note: The majority of respondents value their privacy and do not approve of the government keeping tabs on them.


Version B: "I don't mind if the government keeps tabs on regular people if it helps keep us safer from terrorism."


Results:
Error creating thumbnail: Unable to save thumbnail to destination


Note: Now, the majority of respondents appear to be willing to sacrifice their privacy for perceived safety.

Side by side comparison of Versions A and B:

Error creating thumbnail: Unable to save thumbnail to destination

By creating a frame of "terrorism" for the question, respondents gave a different answer than they otherwise might. Whether you attribute the cause to placing terrorism at the "top of the head" of the respondent<ref>Zaller, John (1992). The Nature and Origins of Mass Opinions. Cambridge University Press. </ref> or causing them to consult a different core predisposition<ref>Alvarez, R. Michael; Brehm, John (2002). Hard Choices, Easy Answers: Values, Information, and American Public Opinion. Princeton University Press. </ref>, the result of altering their response is the same.

Example of Framing Related to Word Choice

In the fall of 2008, the United States Congress was considering The Troubled Asset Relief Program (TARP), the Bush administration's plan to purchase assets and equity from financial institutions in hopes of stabilizing the financial industry. It was signed into law on October 3, 2008. In September 2008, three separate national telephone surveys (Pew, LA Times/Bloomberg, and ABC News/Washington Post) sought to gauge public opinion regarding this program. Each survey used different language to describe the program.

Pew wording: "As you may know, the government is potentially investing billions to try and keep financial institutions and markets secure. Do you think this is the right thing or the wrong thing for the government to be doing?"

Pew results: Right 57%, Wrong 30%

LA Times/Bloomberg wording: "Do you think the government should use taxpayers’ dollars to rescue ailing private financial firms whose collapse could have adverse effects on the economy and market, or is it not the government’s responsibility to bail out private companies with taxpayers’ dollars?"

LA Times/Bloomberg results: Should 31%, Should not 55%

ABC News/Washington Post wording: "Do you approve or disapprove of the steps the Federal Reserve and the Treasury Department have taken to try to deal with the current situation involving the stock market and major financial institutions?"

ABC News/Washington Post results: Approve 44%, Disapprove 42%


Impact of Word Choice on Survey Results
Poll Wording Favor Program Do Not Favor Program
Pew "investing" 57% 30%
LAT/Bloomberg "bail out" 31% 55%
ABC News/Post "steps" 44% 42%


Certainly, there was no intention to bias the results as all three polls have an established national reputation for reliability.  Additionally, at first blush, none of the questions seem to have any obvious problems with wording.  Nonetheless, the results vary dramatically from one poll to the next.  Part of the reason may be the affective connotations attached to the different key words.  "Investing" often has positive connotations.  People think of "investing" as something positive -- storing away something for the future or improving something.  The results of the Pew poll show that a solid majority of Americans liked the idea of "investing" in the security of national markets and financial institutions.  The term "bail out," on the other hand, often has negative connotations.  Many people associate getting bailed out with saving someone from a silly or stupid mistake caused by an action they should not have taken.  It is often seen as undeserved and necessary because of imprudence.  The results of the LA Times/Bloomberg poll show that a definite majority of Americans were against a "bail out" for companies that had gotten themselves in trouble.  Finally, there usually aren't heavy positive or negative connotations with taking "steps."  The language here is much more neutral.  The ABC News/Washington Post poll results show a country evenly divided on the "steps" proposed to "deal with the [...] situation."


It could be that framing from simple differences in word choice is more likely to occur for issues on which individuals do not hold strong opinions or are difficult for an individual to understand.  Any survey question long enough to adequately explain TARP, giving the respondent enough information to fully understand the program, would have been far too long to practically include in a survey.  As such, the respondent may be more likely to look for, or pick up on, inadvertant cues in the question wording.  "Investing" provided a slight positive cue and showed results favorable of TARP.  "Bail out" provided a slight negative cue and showed results unfavorable of TARP.  "Steps" provided no positive or negative cue and show results that could be interpreted to mean that the public was either evenly divided on TARP or were uncertain due to a lack of familiarity with the specifics of TARP.

Priming

Michael Parkin defines priming as the "psychological process in which exposure to a stimulus activates a concept in memory that is then given increased weight in subsequent judgment tasks. Priming works by making the activated concept accessible so that it can be readily used in evaluating related objects."<ref>Parkin, Michael (2008). "Priming". In Paul J. Lavrakas. Encyclopedia of Survey Research Methods. Sage. </ref>

Example of Priming

In surveys, Priming response effects are often the result of question order. Imagine a survey that seeks to measure individuals' concerns about the economy. If respondents are asked what they consider to be the most important problem facing their country or state after they have been asked a series of questions regarding economic concerns, they may be more likely to cite the economy at the "most important problem."

Priming due to question order may also change the criteria used for judgement or evaluation. Asking a series of questions about the economy before asking respondents to evaluate the U.S. president may result in higher evaluations than might otherwise be expected in economic good times and lower than expected evaluations in an economic downturn. Similarly, asking for a presidential evaluation after a series of questions regarding foreign military engagements may result in the individual subconsciously creating their evaluation of the president based on their attitudes regarding foreign policy rather than any domestic issue.

In fact, for these reasons, most political surveys ask questions regarding approval or evaluation for candidates or elected officials, as well as questions regarding the most important problem or issue facing the country or state, somewhere near the beginning of the survey. Another example would be asking questions regarding religious beliefs and church attendance prior to asking questions about issues that are often religiously charged, such as abortion or same sex marriage.

More on Question Order

The order in which questions are asked can impact results in ways other than priming. In a brilliant survey based experiment conducted in 1948 by Herbert Hyman and Paul Sheatsley<ref>Hyman, Herbert; Paul Sheatsley (1950). "The Current Status of American Public Opinion". In J. C. Payne. The Teaching of Contemporary Affairs. Twenty-first Yearbook of the National Council of Social Studies. </ref>, question order was shown to inspire a sense of reciprocity or fairness as it impacted respondent attitudes.

Respondents were asked both of the following questions:

Communist reporter item: "Do you think the United States should let Communist newspaper reporters come in here and send back to their papers the news as they see it?"
American reporter item: "Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?"

However, the order in which the questions were asked was varied. Americans' willingness to allow a Communist reporter into the United States varied greatly depending on which question was asked first.


Question Order Effects
Communist reporter item asked first American reporter item asked first
Percent in favor of allowing a Communist reporter into the United States 36.5% 73.1%


In 1948, as fears surrounding the Cold War grew, individuals who were first asked about allowing a Communist reporter into the United States showed little willingness to open the country to them with only a bit more than one-third favoring admitting the Communist reporter.  However, when asked about allowing in a Communist reporter after being asked whether Communist countries should permit the entrance of an American reporter, more than 7 out of 10 favored allowing in a Communist reporter.  Thinking about Communist countires allowing in an American reporter generated feelings of reciprocal obligation and made respondents more likely to respond favorably to the Communist reporter item.  Howard Schuman and Stanley Presser repeated the experiment exactly in 1980 with nearly as dramatic results.<ref>Template:Cite bookfckLR</ref>


Heading

Sub-heading

Conclusion

References

<references group=""></references>

Discussion questions

Problems

Glossary

  • [[Def: ]]
  • [[Def: ]]
  • [[Def: ]]