Chapter Four: Quantitative Methods (Part 1)

Quantitative Worldview Assumptions: A Review

In chapter 2, you were introduced to the unique assumptions quantitative research holds about knowledge and how it is created, or what the authors referred to in chapter one as "epistemology." Understanding these assumptions can help you better determine whether you need to use quantitative methods for a particular research study in which you are interested.

Quantitative researchers believe there is an objective reality, which can be measured. "Objective" here means that the researcher is not relying on their own perceptions of an event. S/he is attempting to gather "facts" which may be separate from people's feeling or perceptions about the facts. These facts are often conceptualized as "causes" and "effects." When you ask research questions or pose hypotheses with words in them such as "cause," "effect," "difference between," and "predicts," you are operating under assumptions consistent with quantitative methods. The overall goal of quantitative research is to develop generalizations that enable the researcher to better predict, explain, and understand some phenomenon.

Because of trying to prove cause-effect relationships that can be generalized to the population at large, the research process and related procedures are very important for quantitative methods. Research should be consistently and objectively conducted, without bias or error, in order to be considered to be valid (accurate) and reliable (consistent). Perhaps this emphasis on accurate and standardized methods is because the roots of quantitative research are in the natural and physical sciences, both of which have at their base the need to prove hypotheses and theories in order to better understand the world in which we live. When a person goes to a doctor and is prescribed some medicine to treat an illness, that person is glad such research has been done to know what the effects of taking this medicine is on others' bodies, so s/he can trust the doctor's judgment and take the medicines.

As covered in chapters 1 and 2, the questions you are asking should lead you to a certain research method choice. Students sometimes want to avoid doing quantitative research because of fear of math/statistics, but if their questions call for that type of research, they should forge ahead and use it anyway. If a student really wants to understand what the causes or effects are for a particular phenomenon, they need to do quantitative research. If a student is interested in what sorts of things might predict a person's behavior, they need to do quantitative research. If they want to confirm the finding of another researcher, most likely they will need to do quantitative research. If a student wishes to generalize beyond their participant sample to a larger population, they need to be conducting quantitative research.

So, ultimately, your choice of methods really depends on what your research goal is. What do you really want to find out? Do you want to compare two or more groups, look for relationships between certain variables, predict how someone will act or react, or confirm some findings from another study? If so, you want to use quantitative methods.

Self-Quiz

A topic such as self-esteem can be studied in many ways. Listed below are some example RQs about self-esteem. Which of the following research questions should be answered with quantitative methods?

1. Is there a difference between men's and women's level of self- esteem?
2. How do college-aged women describe their ups and downs with self-esteem?
3. How has "self-esteem" been constructed in popular self-help books over time?
4. Is there a relationship between self-esteem levels and communication apprehension?

What are the advantages of approaching a topic like self-esteem using quantitative methods? What are the disadvantages?

For more information, see the following website:
Analyse This!!! Learning to analyse quantitative data

Answers: 1 & 4

Quantitative Methods Part One: Planning Your Study

Planning your study is one of the most important steps in the research process when doing quantitative research. As seen in the diagram below, it involves choosing a topic, writing research questions/hypotheses, and designing your study. Each of these topics will be covered in detail in this section of the chapter.

Topic Choice

Decide on Topic

How do you go about choosing a topic for a research project? One of the best ways to do this is to research something about which you would like to know more. Your communication professors will probably also want you to select something that is related to communication and things you are learning about in other communication classes.

When the authors of this textbook select research topics to study, they choose things that pique their interest for a variety of reasons, sometimes personal and sometimes because they see a need for more research in a particular area. For example, April Chatham-Carpenter studies adoption return trips to China because she has two adopted daughters from China and because there is very little research on this topic for Chinese adoptees and their families; she studied home vs. public schooling because her sister home schools, and at the time she started the study very few researchers had considered the social network implications for home schoolers (cf. http://www.uni.edu/chatham/homeschool.html).

When you are asked in this class and other classes to select a topic to research, think about topics that you have wondered about, that affect you personally, or that know have gaps in the research. Then start writing down questions you would like to know about this topic. These questions will help you decide whether the goal of your study is to understand something better, explain causes and effects of something, gather the perspectives of others on a topic, or look at how language constructs a certain view of reality.

Review Previous Research

In quantitative research, you do not rely on your conclusions to emerge from the data you collect. Rather, you start out looking for certain things based on what the past research has found. This is consistent with what was called in chapter 2 as a deductive approach (Keyton, 2011), which also leads a quantitative researcher to develop a research question or research problem from reviewing a body of literature, with the previous research framing the study that is being done. So, reviewing previous research done on your topic is an important part of the planning of your study. As seen in chapter 3 and the Appendix, to do an adequate literature review, you need to identify portions of your topic that could have been researched in the past. To do that, you select key terms of concepts related to your topic.

Some people use concept maps to help them identify useful search terms for a literature review. For example, see the following website:
Concept Mapping: How to Start Your Term Paper Research.

Narrow Topic to Researchable Area

Once you have selected your topic area and reviewed relevant literature related to your topic, you need to narrow your topic to something that can be researched practically and that will take the research on this topic further. You don't want your research topic to be so broad or large that you are unable to research it. Plus, you want to explain some phenomenon better than has been done before, adding to the literature and theory on a topic. You may want to test out what someone else has found, replicating their study, and therefore building to the body of knowledge already created.

To see how a literature review can be helpful in narrowing your topic, see the following sources. Narrowing or Broadening Your Research Topic and How to Conduct a Literature Review in Social Science

Research Questions & Hypotheses

Write Your Research Questions (RQs) and/or Hypotheses (Hs)

Once you have narrowed your topic based on what you learned from doing your review of literature, you need to formalize your topic area into one or more research questions or hypotheses. If the area you are researching is a relatively new area, and no existing literature or theory can lead you to predict what you might find, then you should write a research question. Take a topic related to social media, for example, which is a relatively new area of study. You might write a research question that asks:

"Is there a difference between how 1st year and 4th year college students use Facebook to communicate with their friends?"

If, however, you are testing out something you think you might find based on the findings of a large amount of previous literature or a well-developed theory, you can write a hypothesis. Researchers often distinguish between null and alternative hypotheses. The alternative hypothesis is what you are trying to test or prove is true, while the null hypothesis assumes that the alternative hypothesis is not true. For example, if the use of Facebook had been studied a great deal, and there were theories that had been developed on the use of it, then you might develop an alternative hypothesis, such as: "First-year students spend more time on using Facebook to communicate with their friends than fourth-year students do." Your null hypothesis, on the other hand, would be: "First-year students do not spend any more time using Facebook to communication with their friends than fourth-year students do." Researchers, however, only state the alternative hypothesis in their studies, and actually call it "hypothesis" rather than "alternative hypothesis."

Process of Writing a Research Question/Hypothesis.

Once you have decided to write a research question (RQ) or hypothesis (H) for your topic, you should go through the following steps to create your RQ or H.

1. Name the concepts from your overall research topic that you are interested in studying.

   RQs and Hs have variables, or concepts that you are interested in studying. Variables can take on different values. For example, in the RQ above, there are at least two variables – year in college and use of Facebook (FB) to communicate. Both of them have a variety of levels within them.

   When you look at the concepts you identified, are there any concepts which seem to be related to each other? For example, in our RQ, we are interested in knowing if there is a difference between first-year students and fourth-year students in their use of FB, meaning that we believe there is some connection between our two variables.

2. If you think the variables you have identified are somehow related to each other, do the following. If not, start your RQ with a word such as "what," "how," "why," or "to what extent," since you are most likely interested in describing something rather than seeing what the connection is between two or more variables or concepts.
   1. Decide what type of a relationship you would like to study between the variables. Do you think one causes the other? Does a difference in one create a difference in the other? As the value of one changes, does the value of the other change?

   2. Identify which one of these concepts is the independent (or predictor) variable, or the concept that is perceived to be the cause of change in the other variable? Which one is the dependent (criterion) variable, or the one that is affected by changes in the independent variable? In the above example RQ, year in school is the independent variable, and amount of time spent on Facebook communicating with friends is the dependent variable. The amount of time spent on Facebook depends on a person's year in school.

      If you're still confused about independent and dependent variables, check out the following site:
      Independent & Dependent Variables.

   3. Express the relationship between the concepts as a single sentence – in either a hypothesis or a research question.

      For example, "is there a difference between international and American students on their perceptions of the basic communication course," where cultural background and perceptions of the course are your two variables. Cultural background would be the independent variable, and perceptions of the course would be your dependent variable. More examples of RQs and Hs are provided in the next section.

      APPLICATION: Try the above steps with your topic now. Check with your instructor to see if s/he would like you to send your topic and RQ/H to him/her via e-mail.

Types of Research Questions/Hypotheses

Once you have written your RQ/H, you need to determine what type of research question or hypothesis it is. This will help you later decide what types of statistics you will need to run to answer your question or test your hypothesis. There are three possible types of questions you might ask, and two possible types of hypotheses. The first type of question cannot be written as a hypothesis, but the second and third types can.

Descriptive Question.

The first type of question is a descriptive question. If you have only one variable or concept you are studying, OR if you are not interested in how the variables you are studying are connected or related to each other, then your question is most likely a descriptive question.

This type of question is the closest to looking like a qualitative question, and often starts with a "what" or "how" or "why" or "to what extent" type of wording. What makes it different from a qualitative research question is that the question will be answered using numbers rather than qualitative analysis. Some examples of a descriptive question, using the topic of social media, include the following.

Notice that neither of these questions has a clear independent or dependent variable, as there is no clear cause or effect being assumed by the question. The question is merely descriptive in nature. It can be answered by summarizing the numbers obtained for each category, such as by providing percentages, averages, or just the raw totals for each type of strategy or organization. This is true also of the following research questions found in a study of online public relations strategies:

The researchers in this study reported statistics in their results or findings section, making it clearly a quantitative study, but without an independent or dependent variable; therefore, these research questions illustrate the first type of RQ, the descriptive question.

Difference Question/Hypothesis.

The second type of question is a question/hypothesis of difference, and will often have the word "difference" as part of the question. The very first research question in this section, asking if there is a difference between 1st year and 4th year college students' use of Facebook, is an example of this type of question. In this type of question, the independent variable is some type of grouping or categories, such as age. Another example of a question of difference is one April asked in her research on home schooling: "Is there a difference between home vs. public schoolers on the size of their social networks?" In this example, the independent variable is home vs. public schooling (a group being compared), and the dependent variable is size of social networks. Hypotheses can also be difference hypotheses, as the following example on the same topic illustrates: "Public schoolers have a larger social network than home schoolers do."

Relationship/Association Question/Hypothesis.

The third type of question is a relationship/association question or hypothesis, and will often have the word "relate" or "relationship" in it, as the following example does: "There is a relationship between number of television ads for a political candidate and how successful that political candidate is in getting elected." Here the independent (or predictor) variable is number of TV ads, and the dependent (or criterion) variable is the success at getting elected. In this type of question, there is no grouping being compared, but rather the independent variable is continuous (ranges from zero to a certain number) in nature. This type of question can be worded as either a hypothesis or as a research question, as stated earlier.

Self-Quiz:

Test out your knowledge of the above information, by answering the following questions about the RQ/H listed below. (Remember, for a descriptive question there are no clear independent & dependent variables.)

• What is the independent variable (IV)?
• What is the dependent variable (DV)?
• What type of research question/hypothesis is it? (descriptive, difference, relationship/association)

1. "Is there a difference on relational satisfaction between those who met their current partner through online dating and those who met their current partner face-to-face?"
2. "How do Fortune 500 firms use focus groups to market new products?"
3. "There is a relationship between age and amount of time spent online using social media."

Answers:
RQ1 is a difference question, with type of dating being the IV and relational satisfaction being the DV.
RQ2 is a descriptive question with no IV or DV.
RQ3 is a relationship hypothesis with age as the IV and amount of time spent online as the DV.

Design Your Study

The third step in planning your research project, after you have decided on your topic/goal and written your research questions/hypotheses, is to design your study which means to decide how to proceed in gathering data to answer your research question or to test your hypothesis. This step includes six things to do. [NOTE: The terms used in this section will be defined as they are used.]

1. Decide type of study design: Experimental, quasi-experimental, non-experimental.
2. Decide kind of data to collect: Survey/interview, observation, already existing data.
3. Operationalize variables into measurable concepts.
4. Determine type of sample: Probability or non-probability.
5. Decide how you will collect your data: face-to-face, via e-mail, an online survey, library research, etc.
6. Pilot test your methods.

Types of Study Designs

With quantitative research being rooted in the scientific method, traditional research is structured in an experimental fashion. This is especially true in the natural sciences, where they try to prove causes and effects on topics such as successful treatments for cancer. For example, the University of Iowa Hospitals and Clinics regularly conduct clinical trials to test for the effectiveness of certain treatments for medical conditions (University of Iowa Hospitals & Clinics: Clinical Trials). They use human participants to conduct such research, regularly recruiting volunteers. However, in communication, true experiments with treatments the researcher controls are less necessary and thus less common. It is important for the researcher to understand which type of study s/he wishes to do, in order to accurately communicate his/her methods to the public when describing the study.

There are three possible types of studies you may choose to do, when embarking on quantitative research: (a) True experiments, (b) quasi-experiments, and (c) non-experiments.

For more information to read on these types of designs, take a look at the following website and related links in it:
Types of Designs.

The following flowchart should help you distinguish between the three types of study designs described below.

True Experiments.

The first two types of study designs use difference questions/hypotheses, as the independent variable for true and quasi-experiments is nominal or categorical (based on categories or groupings), as you have groups that are being compared. As seen in the flowchart above, what distinguishes a true experiment from the other two designs is a concept called "random assignment." Random assignment means that the researcher controls to which group the participants are assigned. April's study of home vs. public schooling was NOT a true experiment, because she could not control which participants were home schooled and which ones were public schooled, and instead relied on already existing groups.

An example of a true experiment reported in a communication journal is a study investigating the effects of using interest-based contemporary examples in a lecture on the history of public relations, in which the researchers had the following two hypotheses: "Lectures utilizing interest- based examples should result in more interested participants" and "Lectures utilizing interest- based examples should result in participants with higher scores on subsequent tests of cognitive recall" (Weber, Corrigan, Fornash, & Neupauer, 2003, p. 118). In this study, the 122 college student participants were randomly assigned by the researchers to one of two lecture video viewing groups: a video lecture with traditional examples and a video with contemporary examples. (To see the results of the study, look it up using your school's library databases).

A second example of a true experiment in communication is a study of the effects of viewing either a dramatic narrative television show vs. a nonnarrative television show about the consequences of an unexpected teen pregnancy. The researchers randomly assigned their 367 undergraduate participants to view one of the two types of shows.

Moyer-Gusé, E., & Nabi, R. L. (2010). Explaining the effects of narrative in an entertainment television program: Overcoming resistance to persuasion. Human Communication Research, 36, 26-52.

A third example of a true experiment done in the field of communication can be found in the following study.

Jensen, J. D. (2008). Scientific uncertainty in news coverage of cancer research: Effects of hedging on scientists' and journalists' credibility. Human Communication Research, 34, 347-369.

In this study, Jakob Jensen had three independent variables. He randomly assigned his 601 participants to 1 of 20 possible conditions, between his three independent variables, which were (a) a hedged vs. not hedged message, (b) the source of the hedging message (research attributed to primary vs. unaffiliated scientists), and (c) specific news story employed (of which he had five randomly selected news stories about cancer research to choose from). Although this study was pretty complex, it does illustrate the true experiment in our field since the participants were randomly assigned to read a particular news story, with certain characteristics.

Quasi-Experiments.

If the researcher is not able to randomly assign participants to one of the treatment groups (or independent variable), but the participants already belong to one of them (e.g., age; home vs. public schooling), then the design is called a quasi-experiment. Here you still have an independent variable with groups, but the participants already belong to a group before the study starts, and the researcher has no control over which group they belong to.

An example of a hypothesis found in a communication study is the following: "Individuals high in trait aggression will enjoy violent content more than nonviolent content, whereas those low in trait aggression will enjoy violent content less than nonviolent content" (Weaver & Wilson, 2009, p. 448). In this study, the researchers could not assign the participants to a high or low trait aggression group since this is a personality characteristic, so this is a quasi-experiment. It does not have any random assignment of participants to the independent variable groups. Read their study, if you would like to, at the following location.

Weaver, A. J., & Wilson, B. J. (2009). The role of graphic and sanitized violence in the enjoyment of television dramas. Human Communication Research, 35 (3), 442-463.

Benoit and Hansen (2004) did not choose to randomly assign participants to groups either, in their study of a national presidential election survey, in which they were looking at differences between debate and non-debate viewers, in terms of several dependent variables, such as which candidate viewers supported. If you are interested in discovering the results of this study, take a look at the following article.

Benoit, W. L., & Hansen, G. J. (2004). Presidential debate watching, issue knowledge, character evaluation, and vote choice. Human Communication Research, 30 (1), 121-144.

Non-Experiments.

The third type of design is the non-experiment. Non-experiments are sometimes called survey designs, because their primary way of collecting data is through surveys. This is not enough to distinguish them from true experiments and quasi-experiments, however, as both of those types of designs may use surveys as well.

What makes a study a non-experiment is that the independent variable is not a grouping or categorical variable. Researchers observe or survey participants in order to describe them as they naturally exist without any experimental intervention. Researchers do not give treatments or observe the effects of a potential natural grouping variable such as age. Descriptive and relationship/association questions are most often used in non-experiments.

Some examples of this type of commonly used design for communication researchers include the following studies.

All of these non-experimental studies have in common no researcher manipulation of an independent variable or even having an independent variable that has natural groups that are being compared.

Data Collection Methods

Once you decide the type of quantitative research design you will be using, you will need to determine which of the following types of data you will collect: (a) survey data, (b) observational data, and/or (c) already existing data, as in library research.

Surveys.

Using the survey data collection method means you will talk to people or survey them about their behaviors, attitudes, perceptions, and demographic characteristics (e.g., biological sex, socio-economic status, race). This type of data usually consists of a series of questions related to the concepts you want to study (i.e., your independent and dependent variables). Both of April's studies on home schooling and on taking adopted children on a return trip back to China used survey data.

On a survey, you can have both closed-ended and open-ended questions. Closed-ended questions, can be written in a variety of forms. Some of the most common response options include the following.

Likert responses – for example: for the following statement, ______ do you
strongly agree agree neutral disagree strongly disagree

Semantic differential – for example: does the following ______ make you
Happy ..................................... Sad

Yes-no answers for example: I use social media daily.
Yes / No.

One site to check out for possible response options is http://www.360degreefeedback.net/media/ResponseScales.pdf.

Researchers often follow up some of their closed-ended questions with an "other" category, in which they ask their participants to "please specify," their response if none of the ones provided are applicable. They may also ask open-ended questions on "why" a participant chose a particular answer or ask participants for more information about a particular topic. If the researcher wants to use the open-ended question responses as part of his/her quantitative study, the answers are usually coded into categories and counted, in terms of the frequency of a certain answer, using a method called content analysis, which will be discussed when we talk about already-existing artifacts as a source of data.

Surveys can be done face-to-face, by telephone, mail, or online. Each of these methods has its own advantages and disadvantages, primarily in the form of the cost in time and money to do the survey. For example, if you want to survey many people, then online survey tools such as surveygizmo.com and surveymonkey.com are very efficient, but not everyone has access to taking a survey on the computer, so you may not get an adequate sample of the population by doing so. Plus you have to decide how you will recruit people to take your online survey, which can be challenging. There are trade-offs with every method.

For more information on things to consider when selecting your survey method, check out the following website:

Selecting the Survey Method.

There are also many good sources for developing a good survey, such as the following websites.
Constructing the Survey
Survey Methods
Designing Surveys

Observation.

A second type of data collection method is observation. In this data collection method, you make observations of the phenomenon you are studying and then code your observations, so that you can count what you are studying. This type of data collection method is often called interaction analysis, if you collect data by observing people's behavior. For example, if you want to study the phenomenon of mall-walking, you could go to a mall and count characteristics of mall-walkers. A researcher in the area of health communication could study the occurrence of humor in an operating room, for example, by coding and counting the use of humor in such a setting.

One extended research study using observational data collection methods, which is cited often in interpersonal communication classes, is John Gottman's research, which started out in what is now called "The Love Lab." In this lab, researchers observe interactions between couples, including physiological symptoms, using coders who look for certain items found to predict relationship problems and success.

Take a look at the YouTube video about "The Love Lab" at the following site to learn more about the potential of using observation in collecting data for a research study: The "Love" Lab.

Already-Existing Artifacts.

The third method of quantitative data collection is the use of already-existing artifacts. With this method, you choose certain artifacts (e.g., newspaper or magazine articles; television programs; webpages) and code their content, resulting in a count of whatever you are studying. With this data collection method, researchers most often use what is called quantitative content analysis. Basically, the researcher counts frequencies of something that occurs in an artifact of study, such as the frequency of times something is mentioned on a webpage. Content analysis can also be used in qualitative research, where a researcher identifies and creates text-based themes but does not do a count of the occurrences of these themes. Content analysis can also be used to take open-ended questions from a survey method, and identify countable themes within the questions.

Content analysis is a very common method used in media studies, given researchers are interested in studying already-existing media artifacts. There are many good sources to illustrate how to do content analysis such as are seen in the box below.

See the following sources for more information on content analysis.
Writing Guide: Content Analysis
A Flowchart for the Typical Process of Content Analysis Research
What is Content Analysis?

With content analysis and any method that you use to code something into categories, one key concept you need to remember is inter-coder or inter-rater reliability, in which there are multiple coders (at least two) trained to code the observations into categories. This check on coding is important because you need to check to make sure that the way you are coding your observations on the open-ended answers is the same way that others would code a particular item. To establish this kind of inter-coder or inter-rater reliability, researchers prepare codebooks (to train their coders on how to code the materials) and coding forms for their coders to use.

To see some examples of actual codebooks used in research, see the following website: Human Coding--Sample Materials.

There are also online inter-coder reliability calculators some researchers use, such as the following: ReCal: reliability calculation for the masses.

Regardless of which method of data collection you choose, you need to decide even more specifically how you will measure the variables in your study, which leads us to the next planning step in the design of a study.

Operationalization of Variables into Measurable Concepts

When you look at your research question/s and/or hypotheses, you should know already what your independent and dependent variables are. Both of these need to be measured in some way. We call that way of measuring operationalizing a variable. One way to think of it is writing a step by step recipe for how you plan to obtain data on this topic. How you choose to operationalize your variable (or write the recipe) is one all-important decision you have to make, which will make or break your study. In quantitative research, you have to measure your variables in a valid (accurate) and reliable (consistent) manner, which we discuss in this section. You also need to determine the level of measurement you will use for your variables, which will help you later decide what statistical tests you need to run to answer your research question/s or test your hypotheses. We will start with the last topic first.

Level of Measurement

Level of measurement has to do with whether you measure your variables using categories or groupings OR whether you measure your variables using a continuous level of measurement (range of numbers). The level of measurement that is considered to be categorical in nature is called nominal, while the levels of measurement considered to be continuous in nature are ordinal, interval, and ratio. The only ones you really need to know are nominal, ordinal, and interval/ratio.

Nominal variables are categories that do not have meaningful numbers attached to them but are broader categories, such as male and female, home schooled and public schooled, Caucasian and African-American. Ordinal variables do have numbers attached to them, in that the numbers are in a certain order, but there are not equal intervals between the numbers (e.g., such as when you rank a group of 5 items from most to least preferred, where 3 might be highly preferred, and 2 hated). Interval/ratio variables have equal intervals between the numbers (e.g., weight, age).

For more information about these levels of measurement, check out one of the following websites.
Levels of Measurement
Measurement Scales in Social Science Research
What is the difference between ordinal, interval and ratio variables? Why should I care?

Validity and Reliability

When developing a scale/measure or survey, you need to be concerned about validity and reliability. Readers of quantitative research expect to see researchers justify their research measures using these two terms in the methods section of an article or paper.

Validity. Validity is the extent to which your scale/measure or survey adequately reflects the full meaning of the concept you are measuring. Does it measure what you say it measures? For example, if researchers wanted to develop a scale to measure "servant leadership," the researchers would have to determine what dimensions of servant leadership they wanted to measure, and then create items which would be valid or accurate measures of these dimensions. If they included items related to a different type of leadership, those items would not be a valid measure of servant leadership. When doing so, the researchers are trying to prove their measure has internal validity. Researchers may also be interested in external validity, but that has to do with how generalizable their study is to a larger population (a topic related to sampling, which we will consider in the next section), and has less to do with the validity of the instrument itself.

There are several types of validity you may read about, including face validity, content validity, criterion-related validity, and construct validity. To learn more about these types of validity, read the information at the following link:
Validity.

To improve the validity of an instrument, researchers need to fully understand the concept they are trying to measure. This means they know the academic literature surrounding that concept well and write several survey questions on each dimension measured, to make sure the full idea of the concept is being measured. For example, Page and Wong (n.d.) identified four dimensions of servant leadership: character, people-orientation, task-orientation, and process-orientation (A Conceptual Framework for Measuring Servant-Leadership). All of these dimensions (and any others identified by other researchers) would need multiple survey items developed if a researcher wanted to create a new scale on servant leadership.

Before you create a new survey, it can be useful to see if one already exists with established validity and reliability. Such measures can be found by seeing what other respected studies have used to measure a concept and then doing a library search to find the scale/measure itself (sometimes found in the reference area of a library in books like those listed below).

Reliability. Reliability is the second criterion you will need to address if you choose to develop your own scale or measure. Reliability is concerned with whether a measurement is consistent and reproducible. If you have ever wondered why, when taking a survey, that a question is asked more than once or very similar questions are asked multiple times, it is because the researchers one concerned with proving their study has reliability. Are you, for example, answering all of the similar questions similarly? If so, the measure/scale may have good reliability or consistency over time.

Researchers can use a variety of ways to show their measure/scale is reliable. See the following websites for explanations of some of these ways, which include methods such as the test-retest method, the split-half method, and inter-coder/rater reliability.
Types of Reliability
Reliability

To understand the relationship between validity and reliability, a nice visual provided below is explained at the following website (Trochim, 2006, para. 2).
Reliability & Validity

Sampling

One of the things you might have critiqued in the previous self-quiz/discussion may have had less to do with the actual survey itself, but rather with how the researchers got their participants or sample. How participants are recruited is just as important to doing a good study as how valid and reliable a survey is.

Imagine that in the article you chose for the last "self-quiz/discussion" you read the following quote from the Pew Research Center's Internet and American Life Project: "One in three teens sends more than 100 text messages a day, or 3000 texts a month" (Lenhart, 2010, para.5). How would you know whether you could trust this finding to be true? Would you compare it to what you know about texting from your own and your friends' experiences? Would you want to know what types of questions people were asked to determine this statistic, or whether the survey the statistic is based on is valid and reliable? Would you want to know what type of people were surveyed for the study? As a critical consumer of research, you should ask all of these types of questions, rather than just accepting such a statement as undisputable fact. For example, if only people shopping at an Apple Store were surveyed, the results might be skewed high.

In particular, related to the topic of this section, you should ask about the sampling method the researchers did. Often, the researchers will provide information related to the sample, stating how many participants were surveyed (in this case 800 teens, aged 12-17, who were a nationally representative sample of the population) and how much the "margin of error" is (in this case +/- 3.8%). Why do they state such things? It is because they know the importance of a sample in making the case for their findings being legitimate and credible. Margin of error is how much we are confident that our findings represent the population at large. The larger the margin of error, the less likely it is that the poll or survey is accurate. Margin of error assumes a 95% confidence level that what we found from our study represents the population at large.

For more information on margin of error, see one of the following websites.
Answers.com Margin of Error
Stats.org Margin of Error
Americanresearchgroup.com Margin of Error
[this last site is a margin of error calculator, which shows that margin of error is directly tied to the size of your sample, in relationship to the size of the population, two concepts we will talk about in the next few paragraphs]

In particular, this section focused on sampling will talk about the following topics: (a) the difference between a population vs. a sample; (b) concepts of error and bias, or "it's all about significance"; (c) probability vs. non-probability sampling; and (d) sample size issues.

Population vs. Sample

When doing quantitative studies, such as the study of cell phone usage among teens, you are never able to survey the entire population of teenagers, so you survey a portion of the population. If you study every member of a population, then you are conducting a census such as the United States Government does every 10 years. When, however, this is not possible (because you do not have the money the U.S. government has!), you attempt to get as good a sample as possible.

Characteristics of a population are summarized in numerical form, and technically these numbers are called parameters. However, numbers which summarize the characteristics of a sample are called statistics.

Error and Bias

If a sample is not done well, then you may not have confidence in how the study's results can be generalized to the population from which the sample was taken. Your confidence level is often stated as the margin of error of the survey. As noted earlier, a study's margin of error refers to the degree to which a sample differs from the total population you are studying. In the Pew survey, they had a margin of error of +/- 3.8%. So, for example, when the Pew survey said 33% of teens send more than 100 texts a day, the margin of error means they were 95% sure that 29.2% - 36.8% of teens send this many texts a day.

Margin of error is tied to sampling error, which is how much difference there is between your sample's results and what would have been obtained if you had surveyed the whole population. Sample error is linked to a very important concept for quantitative researchers, which is the notion of significance. Here, significance does not refer to whether some finding is morally or practically significant, it refers to whether a finding is statistically significant, meaning the findings are not due to chance but actually represent something that is found in the population. Statistical significance is about how much you, as the researcher, are willing to risk saying you found something important and be wrong.

For the difference between statistical significance and practical significance, see the following YouTube video: Statistical and Practical Significance.

Scientists set certain arbitrary standards based on the probability they could be wrong in reporting their findings. These are called significance levels and are commonly reported in the literature as p <.05 or p <.01 or some other probability (or p) level.

If an article says a statistical test reported that p < .05, it simply means that they are most likely correct in what they are saying, but there is a 5% chance they could be wrong and not find the same results in the population. If p < .01, then there would be only a 1% chance they were wrong and would not find the same results in the population. The lower the probability level, the more certain the results.

When researchers are wrong, or make that kind of decision error, it often implies that either (a) their sample was biased and was not representative of the true population in some way, or (b) that something they did in collecting the data biased the results. There are actually two kinds of sampling error talked about in quantitative research: Type I and Type II error. Type 1 error is what happens when you think you found something statistically significant and claim there is a significant difference or relationship, when there really is not in the actual population. So there is something about your sample that made you find something that is not in the actual population. (Type I error is the same as the probability level, or .05, if using the traditional p-level accepted by most researchers.) Type II error happens when you don't find a statistically significant difference or relationship, yet there actually is one in the population at large, so once again, your sample is not representative of the population.

For more information on these two types of error, check out the following websites.
Hypothesis Testing: Type I Error, Type II Error
Type I and Type II Errors - Making Mistakes in the Justice System

Researchers want to select a sample that is representative of the population in order to reduce the likelihood of having a sample that is biased. There are two types of bias particularly troublesome for researchers, in terms of sampling error. The first type is selection bias, in which each person in the population does not have an equal chance to be chosen for the sample, which happens frequently in communication studies, because we often rely on convenience samples (whoever we can get to complete our surveys). The second type of bias is response bias, in which those who volunteer for a study have different characteristics than those who did not volunteer for the study, another common challenge for communication researchers. Volunteers for a study may very well be different from persons who choose not to volunteer for a study, so that you have a biased sample by relying just on volunteers, which is not representative of the population from which you are trying to sample.

Probability vs. Non-Probability Sampling

One of the best ways to lower your sampling error and reduce the possibility of bias is to do probability or random sampling. This means that every person in the population has an equal chance of being selected to be in your sample. Another way of looking at this is to attempt to get a representative sample, so that the characteristics of your sample closely approximate those of the population. A sample needs to contain essentially the same variations that exist in the population, if possible, especially on the variables or elements that are most important to you (e.g., age, biological sex, race, level of education, socio-economic class).

There are many different ways to draw a probability/random sample from the population. Some of the most common are a simple random sample, where you use a random numbers table or random number generator to select your sample from the population.

There are several examples of random number generators available online. See the following example of an online random number generator: http://www.randomizer.org/.

A systematic random sample takes every n-th number from the population, depending on how many people you would like to have in your sample. A stratified random sample does random sampling within groups, and a multi-stage or cluster sample is used when there are multiple groups within a large area and a large population, and the researcher does random sampling in stages.

If you are interested in understanding more about these types of probability/random samples, take a look at the following website:
Probability Sampling.

However, many times communication researchers use whoever they can find to participate in their study, such as college students in their classes since these people are easily accessible. Many of the studies in interpersonal communication and relationship development, for example, used this type of sample. This is called a convenience sample. In doing so, they are using a non- probability or non-random sample. In these types of samples, each member of the population does not have an equal opportunity to be selected. For example, if you decide to ask your facebook friends to participate in an online survey you created about how college students in the U.S. use cell phones to text, you are using a non-random type of sample. You are unable to randomly sample the whole population in the U.S. of college students who text, so you attempt to find participants more conveniently. Some common non-random or non-probability samples are:

• accidental/convenience samples, such as the facebook example illustrates
• quota samples, in which you do convenience samples within subgroups of the population, such as biological sex, looking for a certain number of participants in each group being compared
• snowball or network sampling, where you ask current participants to send your survey onto their friends.

For more information on non-probability sampling, see the following website:
Nonprobability Sampling.

Researchers, such as communication scholars, often use these types of samples because of the nature of their research. Most research designs used in communication are not true experiments, such as would be required in the medical field where they are trying to prove some cause-effect relationship to cure or alleviate symptoms of a disease. Most communication scholars recognize that human behavior in communication situations is much less predictable, so they do not adhere to the strictest possible worldview related to quantitative methods and are less concerned with having to use probability sampling.

They do recognize, however, that with either probability or non-probability sampling, there is still the possibility of bias and error, although much less with probability sampling. That is why all quantitative researchers, regardless of field, will report statistical significance levels if they are interested in generalizing from their sample to the population at large, to let the readers of their work know how confident they are in their results.

Size of Sample

The larger the sample, the more likely the sample is going to be representative of the population. If there is a lot of variability in the population (e.g., lots of different ethnic groups in the population), a researcher will need a larger sample. If you are interested in detecting small possible differences (e.g., in a close political race), you need a larger sample. However, the bigger your population, the less you have to increase the size of your sample in order to have an adequate sample, as is illustrated by an example sample size calculator such as can be found at http://www.raosoft.com/samplesize.html.

Using the example sample size calculator, see how you might determine how large of a sample you might need in order to study how college students in the U.S. use texting on their cell phones. You would have to first determine approximately how many college students are in the U.S. According to ANEKI, there are a little over 14,000,000 college students in the U.S. (Countries with the Most University Students). When inputting that figure into the sample size calculator below (using no commas for the population size), you would need a sample size of approximately 385 students. If the population size was 20,000, you would need a sample of 377 students. If the population was only 2,000, you would need a sample of 323. For a population of 500, you would need a sample of 218.

It is not enough, however, to just have an adequate or large sample. If there is bias in the sampling, you can have a very bad large sample, one that also does not represent the population at large. So, having an unbiased sample is even more important than having a large sample.

So, what do you do, if you cannot reasonably conduct a probability or random sample? You run statistics which report significance levels, and you report the limitations of your sample in the discussion section of your paper/article.

Pilot Testing Methods

Now that we have talked about the different elements of your study design, you should try out your methods by doing a pilot test of some kind. This means that you try out your procedures with someone to try to catch any mistakes in your design before you start collecting data from actual participants in your study. This will save you time and money in the long run, along with unneeded angst over mistakes you made in your design during data collection. There are several ways you might do this.

You might ask an expert who knows about this topic (such as a faculty member) to try out your experiment or survey and provide feedback on what they think of your design. You might ask some participants who are like your potential sample to take your survey or be a part of your pilot test; then you could ask them which parts were confusing or needed revising. You might have potential participants explain to you what they think your questions mean, to see if they are interpreting them like you intended, or if you need to make some questions clearer.

The main thing is that you do not just assume your methods will work or are the best type of methods to use until you try them out with someone. As you write up your study, in your methods section of your paper, you can then talk about what you did to change your study based on the pilot study you did.

Institutional Review Board (IRB) Approval

The last step of your planning takes place when you take the necessary steps to get your study approved by your institution's review board. As you read in chapter 3, this step is important if you are planning on using the data or results from your study beyond just the requirements for your class project. See chapter 3 for more information on the procedures involved in this step.

Conclusion: Study Design Planning

Once you have decided what topic you want to study, you plan your study. Part 1 of this chapter has covered the following steps you need to follow in this planning process:

1. decide what type of study you will do (i.e., experimental, quasi-experimental, non- experimental);
2. decide on what data collection method you will use (i.e., survey, observation, or already existing data);
3. operationalize your variables into measureable concepts;
4. determine what type of sample you will use (probability or non-probability);
5. pilot test your methods; and
6. get IRB approval.

At that point, you are ready to commence collecting your data, which is the topic of the next section in this chapter.