Actions

Logic of scientific inquiry

From OPOSSEM


Objectives[edit]

Students who complete this unit will be able to

  • identify the primary steps of the scientific method
  • distinguish between inductive and deductive reasoning
  • evaluate research questions and hypotheses
  • distinguish between dependent and independent variables
  • appreciate the importance of the accumulation of scientific knowledge

Introduction[edit]

Science provides a framework for exploring and understanding the world in which we live. While people tend to associate "science" with the natural sciences (biology, chemistry, physics, geology, etc.), the principles of scientific inquiry can be used to understand social phenomena, such as culture, economics, and politics.

What exactly is scientific inquiry? What does it mean to approach a problem or question in a scientific manner? In brief, scientific inquiry is a process of asking and answering questions in an ongoing dialogue about the nature of the world around us. Through systematic questioning, testing, and revising, the scientific community moves closer and closer to a shared understanding of "the truth."

Theories and Hypotheses[edit]

Theories and hypotheses are the currency of scientific inquiry. Before we delve into the logic of how this process works, it is worth spending a moment on what these terms mean and how they differ from one another.

Scholars deal with ideas ranging from the very abstract to the very concrete. Ultimately, they are interested in the relationships between these ideas. When scholars discuss the general relationships between abstract concepts, they often refer to theories. When they discuss more specific relationships between more concrete, observable ideas, they refer to hypotheses. For students and scholars alike, the line between theory and hypothesis can be quite confusing.

Let's consider a relationship which is definitely abstract and general enough to constitute a theory: the rational choice theory of human behavior. Rational choice theory assumes that individuals have preferences about how they would like the world to look. For example, Joe may prefer to have peanut butter for lunch rather than bologna, a Democrat in the White House rather than a Republican, and more money in his bank account rather than less. Given those preferences (and constraints such as knowledge and resources), rational choice theory states that Joe's actions will be designed to achieve a state of the world he most prefers. That is a very broad statement about human behavior and the relationship between desires and actions. It is a theory.

On the other end of the spectrum are hypotheses: specific predictions about the relationships between more concrete aspects of the world around us. For example, the rational choice theory, if true, leads us to a number of hypotheses. One such hypothesis is that ideologically liberal judges will be more likely to resolve Fourth Amendment issues in favor of the individual criminal defendant (as opposed to ruling in favor of the police). This is a much more specific statement about an expected relationship (in this case, the relationship between judicial ideology and search and seizure decisions).

We could state a number of similar expected relationships at various levels of abstraction in between our rational choice theory and our very specific hypothesis about search and seizure cases. It may be difficult to determine, at those intermediate levels of abstraction, whether a statement constitutes a theory or a hypothesis. Students and scholars should not be overly concerned with the semantics here. Scientific inquiry is about stating and testing these possible relationships, no matter what we call them.

Asking Questions[edit]

Getting down to brass tacks, the first step in scientific inquiry is developing a research question.

Most students (and, indeed, most professional scholars) are inspired by interest in a general subject. For example, a political scientist might be interested in the U.S. Supreme Court. She might even set out to write a book describing the Court and its procedures. This, however, is not scientific inquiry.

Scientific inquiry doesn’t begin until she inquires . . . she formulates a question that can be answered using empirical evidence. How does the Supreme Court select which cases it will hear? How do justices decide on the merits of the cases before them? How do Supreme Court decisions affect public opinion? How does public opinion affect Supreme Court decisions? These types of questions form the impetus for scientific inquiry.

Which questions we ask and the way in which we ask them will determine the type of knowledge our inquiry generates, so we should never discount the importance of this first step. You may have been told at various points in your life that there is no such thing as a bad question. This is almost true. However, in the context of scientific inquiry, some questions are definitely better than others.

Good scientific research questions have a satisfactory “so what?” While scholars are usually inspired in the first instance by their innate curiosity, the purpose of scientific inquiry is not merely to satisfy that curiosity. Rather, scientific inquiry should contribute to a broader body of shared knowledge. It should have application beyond a single instance.

For example, our hypothetical scholar of the Supreme Court might pose a very specific question: Why did Justice Douglas retire when he did? The answer to that question might be interesting to the researcher, but the answer will likely be little more than a historical anecdote.

In crafting good research questions, scholars have to be looking ahead to the answer to the questions and the inferences that can be drawn from those answers. If we knew why Justice Douglas retired when he did, what could we do with that information? Could we use it to predict when other justices might retire? Could we use it to understand better the motivations for justices taking a position on the Supreme Court? Or the motivations that influence their decisions while they are serving? In order to craft a more generally relevant question, our researcher needs to draw back from the specific instance: How do justices decide to retire? What motivates individuals to serve on the Supreme Court? What factors influence judicial decision-making more generally?

Answering Questions: The Scientific Method[edit]

Once a researcher has asked a question, she must attempt to answer that question. She does so through a process called the scientific method. As the name suggests, the scientific method is methodical; it is a fairly specific way of addressing an issue, question, or problem. Although the precise roadmap for the scientific method varies a bit from discipline to discipline (and even from researcher to researcher, it is useful to think of the scientific method as a four-step process: (1) observation; (2) generating hypotheses; (3 testing those hypotheses through observation; and (4) refining the hypotheses in light of these new observations.<ref>Gravetter, Frederkick J.; Forzano, Lori-Ann (2009). Research Methods for the Behavioral Sciences (3rd ed.). Wadsworth. pp. 16-22. </ref> By following these steps and documenting the process, scientists and scholars create a path that can be retraced by future researchers.

Observation[edit]

The first step in the scientific method is observation. Many social scientists engage in observation by engaging in field work. They travel to foreign countries, interview political decision-makers, observe legislative and courtroom proceedings, and attend party conventions. Even more social scientists engage in observation by following the news. Social scientists also employ indirect observation by reading the work of other scholars and participating in scholarly workshops and conferences. The fruit of all of these activities is knowledge about political processes.

Generating Hypotheses[edit]

Armed with a research question and a good working knowledge of the political process, a researcher begins to consider possible answers to that question. We call this process hypothesis formulation or hypothesis generation. A hypothesis is simply a prediction (an informed guess) about the causal relationship between two or more phenomena.

Turning our attention back to our hypothetical scholar of the Courts, let’s assume that she chose “What factors affect a justice’s decision to retire?” as her research question. She now must think of possible answers to her question. There are two general ways she might go about generating hypotheses: induction and deduction.

Induction involves looking at specific events in the real world and attempting to generalize from them. For example, our Courts scholar might read biographies of Supreme Court justices and notice that many justices appointed by Democratic presidents seem to retire when a Democrat is in the White House. From these real world observations, she infers that there might be a relationship between partisanship and the decision to retire.

Deduction involves drawing on a body of theoretical knowledge and making specific predictions about the real world. Instead of reading judicial biographies, our Courts scholar may familiarize herself with existing scholarship about judicial motivations and behavior. She may encounter a general theory (called the Attitudinal Model<ref>Segal, Jeffrey A.; Spaeth, Harold (2002). The Supreme Court and the Attitudinal Model Revisited. Cambridge University Press. ISBN 978-0521783514. </ref>) that posits that Supreme Court justices’ behavior is motivated by ideological objectives. Assuming that theory is true, our scholar might reason that justices would be concerned about who would replace them when they retire and thus might wait to retire until the president (who will choose their successor) is someone who shares their ideological beliefs.

In other words, inductive reasoning involves generalizing from specific observations. Deductive reasoning involves making specific predictions based on generalized theory. Note that both inductive reasoning and deductive reasoning might lead our scholar to the same hypothesis: Justices are more likely to retire during the term of an ideologically similar president. In reality, the process of developing hypotheses is usually some blend of inductive and deductive reasoning. Our scholar might be familiar with the attitudinal model of judicial decision making but only draws the connection between ideology and retirement when she reads about Democrat-appointed Justice Douglas’s deep reluctance to leave the Court—-despite serious health concerns—-during the tenure of a Republican president.

Just as all research questions are not created equal, so too there are good hypotheses and bad hypotheses. A good hypothesis should possess a number of characteristics: (1) it should be positive (as opposed to normative); (2) it should be specific; (3) it should be plausible; and (4) it should be testable.<ref>Johnson, Janet Buttolph; Reynolds, H.T.; Jason D. Mycoff (2008). Political Science Research Methods (Sixth ed.). CQ Press. pp. 70-77. ISBN 978-0872894426. </ref>

Positivity[edit]

Scientists are people. They possess values and morals, and many of them have real normative concerns that underlie their research interests. For example, our Courts scholar may have strong beliefs about how the courts and judges should behave, in an ideal world, to achieve just and equitable outcomes.

Scientific inquiry, however, does not attempt to answer these normative questions. The goal of science is not to determine how the world ought to be, but rather to describe how the world actually is.

The formulation of hypotheses should reflect this distinction and this focus. A hypothesis should be a prediction about an observable relationship, rather than an evaluation or judgment about that relationship. For example, “Democratic-appointed justices make better decisions about retirement than do Republican-appointed justices” is a poor hypothesis. What do we mean by better? The word itself presupposes some sort of normative evaluation about what kind of decision justices ought to make. Moreover, this sort of statement cannot be falsified. In other words, there is no way to prove that this statement is false, because any determination of falsity will depend on the subjective determination of what we mean by "better." In contrast, “Democratic-appointed justices are more likely to consider presidential partisanship in retirement decisions” is a purely positive hypothesis. Using empirical observations about actual justices and their actual decisions, we may find that this statement is simply untrue (in other words, this statement is falsifiable).

Specificity[edit]

Well-formulated scientific hypotheses should be quite specific about what the researcher expects to find. A well-formulated hypothesis should specify exactly what phenomena the researcher will observe (the variables); it should imply a causal relationship between the variables; and it should predict the direction of that relationship.

Our Supreme Court researcher, for example, cannot simply state that Supreme Court justices consider political factors in making retirement decisions. While that statement may prove to be true, it lacks the specificity of a good scientific hypothesis. Our Courts scholar needs to be more precise about what she means by “political factors.” For example, is she predicting a relationship between judicial retirement decisions and the party of the sitting president? Or the partisan control of the U.S. Senate? Or the condition of divided government (when Congress is controlled by one party and the White House is controlled by the other party)?

It is not generally enough that two variables seem to move together. A good hypothesis should specify and predict that a change in one variable (the independent variable) causes a change in another (the dependent variable), and it should predict what kind of effect the causal variable has on the other variable. For example, if our researcher predicts a relationship between retirement decisions and the condition of divided government, does she expect that divided government causes an increase in judicial retirements? Or that an increase in judicial retirements causes divided government? The latter doesn't make much sense, so she probably predicts the former; under those circumstances, divided government is the independent variable and judicial retirements is the dependent variable (because judicial retirements depend on divided government). Her hypothesis should also specify whether she predicts a positive relationship (justices are more likely to retire when there is divided government) or a negative relationship (justices are less likely to retire when there is divided government).

Plausibility[edit]

There’s an old saying that if it looks like a duck and quacks like a duck, it’s probably a duck. In the context of scientific inquiry, however, just because a statement looks like a hypothesis, that doesn’t mean it is a plausible hypothesis. A plausible hypothesis should presuppose a legitimate causal mechanism. In other words, the researcher should be able to explain why she expects phenomenon A to affect phenomenon B.

For example, the statement “Justices are more likely to retire on sunny days” looks like a scientific hypothesis. But it is hard to imagine why a Supreme Court justice would care about the weather when he decides to retire. Moreover, justices usually make the decision to retire weeks, if not months, before they actually announce their retirement. In short, a reasonable person would not expect to find empirical support for this “hypothesis” . . . and if they happened to find an apparent relationship between the weather and retirement decisions, they would chalk it up to a fluke.

Ultimately, a scholar will need to rely on empirical observations to support or falsify his hypotheses. However, even before the scholar reaches the point of testing his hypothesis, it should appear plausible. It is not enough to develop a statement that looks like a credible hypothesis. Rather, legitimate scientific inquiry proceeds on the basis of hypotheses that have a reasonable likelihood of being confirmed.

Testability[edit]

Testability is a much more practical concern in the development of workable scientific hypotheses. Scholars do not generate hypotheses just for the heck of it. They do so with the intent, ultimately, of determining whether empirical observation supports or refutes those hypotheses. It makes little sense, then, to develop hypotheses that cannot be tested against empirical observations.

For example, our Supreme Court scholar may hypothesize that Supreme Court justices are more likely to retire when they are bored with their jobs. Her hypothesis, then, is that job boredom (a phenomenon that psychologists test using a survey instrument) positively affects judicial retirements. This is a positive, specific, and plausible hypothesis. But how can we test it? We are unlikely to convince justices of the Supreme Court to take a psychological test to measure their boredom with their job, and it would be difficult (if not impossible) to guess at their level of boredom from their observable behavior. In short, this is a hypothesis that we cannot test.

Hypothesis Testing[edit]

At this point, our researcher has a question she is interested in answering, and she has a possible answer (her hypothesis) in the form of a prediction about how one feature of the real world (an independent variable) will affect another feature of the real world (a dependent variable). Her next step is to test whether the real world supports that prediction.

We call this process "hypothesis testing," and it is covered in greater detail in the specific module on hypothesis testing. For now, it enough to note that the process of testing a hypothesis involves further observation of the real world. We refer to this as "data collection." Social scientists collect data by conducting interviews and surveys, consulting public records (such as court filings or legislative votes), and conducting experiments on human subjects. The goal of all of these activities is to acquire information about multiple events that can be compared and contrasted.

For example, a comparative political scholar might gather information on several countries to see whether countries that have certain characteristics behave differently than countries that lack those characteristic. A political psychologist interested in how TV news affects political judgment might do a controlled experiment where some individuals are shown a news program while others are shown an episode of Friends, and then the researcher measures their opinions about the president to see if there are differences between the two groups. Our courts scholar might go through historical public records to collect information about when and under what political conditions various justices have retired.

While there are many methods for observing the world for purposes of testing hypotheses, all methods of scientific observation must be systematic. Scientific researchers should use the most reliable sources for information; for example, a comparative scholar looking for information about public opinion in England should rely on official government reports or surveys conducted by reliable scholars, rather than referring to old episodes of Fawlty Towers on BBC America. Moreover, if a researcher conducts interviews of multiple people, she should ask the same or very similar questions of each person so that the responses given can be directly compared.

Reporting and Refining[edit]

After testing one or more hypotheses, a researcher will take the knowledge acquired through these new observations and incorporate them into her understanding of the world. She will use this updated worldview to generate new hypotheses.

For example, let us consider our Supreme Court scholar. Perhaps she started her scientific journey with her interest in Justice Douglas's decision to retire, and, informed by that observation and the broader theory about judicial decision-making, she developed the hypothesis that justices are more likely to retire when the sitting president is of the same party as the president who appointed the justices. She gathers additional data about presidential parties and retirement dates for many more Supreme Court justices. She analyzes this data, and she fails to support her hypothesis. She must reject it.

Her inquiry, though, is probably not over. Now she will look at her work to see where she might have gone wrong.

  1. Perhaps the problem was with the hypothesis itself. Perhaps justices are not more likely to retire when the president is of the same party as the justice's appointing president. This would be an interesting finding because, to the extent that the researcher's original hypothesis flowed logically from well-established theory, this "null finding" might suggest an important exception to the general theory. Maybe justices are not quite as devoted to ideological outcomes as existing theory would suggest.
  2. Perhaps the problem is with the assumptions that inform the hypothesis. In other words, perhaps justices are concerned with ideology when they make retirement decisions, but their decision-making process is more complex than the hypothesis assumes. For example, maybe justices are concerned about the party of the president, but only under certain conditions. If a justice is part of a very small minority or a very large minority, the ideology of his replacement will have less of an effect on the overall composition of the Supreme Court. So, perhaps, the hypothesis is true but only for certain justices (those who are members of Supreme Courts that are sharply ideologically divided).
  3. Perhaps the problem is with the specification of variables or the statistical analysis used by the researcher. Maybe the underlying hypothesis is correct, but the researcher has not looked for the correct real-world representations of the concepts she is studying. Our researcher chose to measure the justice's personal ideology by observing the party of the president who appointed the justice. However, perhaps that is not a good measure. Perhaps, instead, the researcher should be looking at the justice's opinions and speeches to determine whether the justice is liberal or conservative. If so, the researcher would want to revise her hypothesis to state that a liberal justice is more likely to retire when the sitting president is a Democrat.

No matter what kind of results our researcher achieved--whether her analysis supported or refuted her hypothesis--she needs to be able to document how she developed her hypotheses, how she measured the concepts she was studying, how she analyzed the data she collected, and her rationale for supporting or rejecting her hypothesis. This will allow her colleagues to challenge and verify her findings, and it will allow her to share her results in a meaningful dialogue with the rest of the scientific community.

The Cooperative Nature of Scientific Research[edit]

Scientists--including social scientists--do not operate in a vacuum. Rather, they work as part of a broader community that engages in peer review, debate, and replication.

When a political science researcher empirically tests her hypothesis, she generally prepares a written report of what she did, why she did it, and how it worked out. In order to get this written record published in a reputable journal, she must subject it to peer review; other scholars who are familiar with her area of research will read the paper (anonymously) and identify potential problems with her methods and her conclusions. She may also present her research orally at a conference, again allowing other researchers to comment on and criticize her work. This may sound like an unpleasant prospect; indeed, for many researchers it is frustrating to have other people dissect their logic and question their assumptions. Yet peer review and debate are critical to the scientific process. It helps to ensure that the research which ultimately makes it into scientific journals is more reliable (not perfect, mind you, but more reliable). Moreover, this critical interaction may spark ideas for new lines of inquiry, keeping the discipline moving forward and acquiring ever more comprehensive knowledge.

Another mechanism the scientific community uses to verify the reliability of research results is replication. When a scholar collects data--observations--to test her hypotheses, she will often make that data available to other researchers. They can look under the hood, so to speak, and recreate the researcher's tests of that data to verify that her methods of analysis are correct. In addition, other researchers will recreate her empirical tests using other observations in order to verify that her conclusions are not simply an artifact of her specific observations. In other words, they will verify that her conclusions can be generalized beyond her specific observations and represent legitimate inferences about the broader world.

Again, it may seem like submitting your hard work to such critical analysis would be a painful process. Sometimes it is. But every researcher who undertakes scientific inquiry is part of an endeavor much larger than herself. She is contributing to the collective body of scientific knowledge (sometimes called the "scholarship"). By working together, sharing data, and critiquing each others' work, researchers are able to make better, more reliable contributions to that collective body of knowledge.

Conclusion[edit]

Science is an iterative, collective endeavor. Science is never "done." Instead, each scientific scholar is building on the work of scholars who came before them, and producing results which can be tested, replicated, and challenged by scholars in the future. The accumulation of this work creates a body of knowledge which becomes more and more reliable over time.

References[edit]

<references group=""></references>

Discussion questions[edit]

Problems[edit]

Glossary[edit]

  • [[Def: ]]
  • [[Def: ]]
  • [[Def: ]]