Out of the Box Coaching and
Breakthroughs with the Enneagram, Mary R. Bast, Ph.D. 
Copyright © 1999. All rights reserved. Revised: December 31, 2011  

Click on "Contact" below left to send email   

Follow My Blogs:  ► Self-Coaching Tips    ► Coach Mentor

Social Research: Guides to a Decision-Making Process
by Susan Gustavus Philliber, Mary (Schwab) Bast, and G. Sam Sloss, copyright © 1980, Peacock Publishers, Inc.

Chapter 5: Data Collection

The sources of data for any study can be infinitely varied. In this chapter we discuss the advantages and disadvantages of four major data collection strategies: questionnaires, interviews, observations, and available data.

QUESTIONNAIRES AND INTERVIEWS

Questionnaires and interviews are the most widely used techniques in the social sciences to collect data, in some disciplines up to 90 percent. National organizations such as the U.S. Bureau of the Census and commercial institutions such as the Gallup Poll also use these techniques. Questionnaires and interviews are routinely used as part of ongoing situations in both business and academic settings.

While these sources of data are frequently used in surveys, they're also used in experiments and case studies. For example, as part of a pre-test or post-test an experimenter might use a questionnaire to solicit background information or measure attitudes. In a case study of the hierarchy of a university, interview schedules might be used to obtain prestige rankings of faculty or administrators. Regardless of your study design, there are several important factors to consider in making effective use of questionnaires and interviews.

The Setting

The settings where questionnaires are filled out or interviews conducted may vary widely, and it's important to consider their impact on your particular study. One of the most popular settings is the respondent's residence. Quite likely you've received a questionnaire in the mail, answered a phone call requesting an interview, or even had an interviewer appear at your door. The setting could also be a street corner, parking lot, or grocery store, where an interviewer might seek opinions on a political candidate, a brand of soap, or a neighborhood project. Surveys can be conducted through internet access, using such formats as Survey Monkey. At other times questionnaires or interviews are used in a more "captive" environment, such as the classroom or work setting, where a number of people may fill out questionnaires simultaneously rather than individually.

The setting is, of course, largely dependent on your sampling procedures. While this has obvious implications for generalization, the setting also has implications for validity. Answers to questions about sexual behavior, for example, are unlikely to be as candid on the street corner as they would be in a more private setting. Similarly, respondents are likely to be more attentive to filling out a questionnaire over breakfast and morning coffee than while watching a favorite TV program. Complex responses to difficult questions are more likely to be facilitated by the physical presence of an interviewer as compared to an interview over the phone.

Introducing the Research

The first task of the researcher is to elicit the cooperation of potential respondents. When questionnaires or interviews are group-administered in a classroom or work situation with teacher or boss involvement, cooperation may not be much of a problem. However, with mailed, e-mailed, or door-to-door interviews there is always the possibility that people will refuse. Respondents are likely to believe the motive for questions is sale of a product or interference with privacy for some secret purpose. Particularly in interviewing, potential respondents are increasingly cautious about talking to strangers or sharing private information.

It's thus important to begin the questionnaire with a cover letter or the interview with a personal introduction. This introduction should establish the legitimacy of the research and the institution sponsoring the project, as well as the credentials of the researcher. In the case of interviews it's often helpful to have a letter from a mutual contact or public official showing the research's approval by an ethics committee or other legitimized body. It may also be good to notify the police and carry identification with you.

Once introduced, the study should be explained in such a way that the respondent will be interested in participating. In addition, respondents should be told how they were chosen to take part in the study. It's not necessary to detail the intricacies of your sampling procedure, but only to assure them they weren't singled out in some way that might make them hesitant to cooperate. Generally, it's sufficient to assure them sampling was random. As part of this explanation, it's useful to assure respondents of confidentiality. They may be told that only group totals will be examined, and no individuals signaled out in the study. Indeed, if it's not necessary to attach names or addresses to questionnaires or interviews, respondent confidence is likely to be increased. Questionnaire or interview forms can still be numbered to assure adequate recordkeeping and to check the characteristics of those who fail to respond.

In the interview, probably the most important task is to establish rapport. While the interviewer may possess some information about the respondent, the interview situation is often an interaction between strangers, and people are sometimes reluctant to allow strangers to ask them questions. Thus, it's important to have a friendly and neutral manner that expresses acceptance without being either overly sympathetic or critical. The use of familiar language tends to set respondents at ease. Experience has shown, however, that it's best to be authentic, using language that's comfortable for you. Imitating their colloquialisms can be offensive, and people respond more readily to someone with an empathic and understanding manner than to someone they perceive as phony.

Instrument Format

During the earlier stages of the research project you decide what questions or operational measures to use in the study. However, the order and sequence of presenting these measures can have consequences for the way questions are answered. When questionnaires are used, basic considerations include neatness and orderliness. The questionnaire should be easy to follow and unambiguous in terms of where answers should be placed.

Such things as readability should be considered. Is the print large enough? Is there adequate spacing between lines? Are instructions clear? Are questions printed on both sides of each sheet and, if so, will respondents miss some questions? It's often helpful to use the same method for response throughout; for example, circling, underlying, or using checkmarks. In addition, the length of the questionnaire or interview should be considered. Generally people will more readily answer a short instrument, but if the research necessitates a longer one, it might be made to appear shorter by reducing the size of the print or by using both sides of the paper.

One factor that may affect the length of a questionnaire is the use of penalty branches. For example, the respondent might be instructed: "If the answer to the preceding question was 'Yes,' please answer the following 10 questions; if the answer was 'No,' go on to question 34." As a "penalty" for answering "Yes" they must answer ten additional questions. Instead, the penalty might be placed at a point in the questionnaire where it would be difficult to go back and change an answer, or placed so as to obtain an answer to the initial question before the penalty is obvious. Clearly, some concessions may have to be made in weighing these practical considerations, but it's important to save the respondent as much time and effort as possible while still getting honest replies.

Also pay attention to question order. If the questionnaire or interview contains sensitive questions, you'll want to ease into them. You may even begin such sections with reminders of anonymity. At the same time, the questioning shouldn't begin with a long list of boring queries that make the research seem routine and full. Respondents are more cooperative with interesting questionnaires or interviews.

Another implication of question order is that early questions may affect later answers. For example, suppose you wish to know what people consider the most important social problems. You also wish to know how they feel about specific social problems. In this case, first ask the more general question about what they consider to be problems before giving them a list of such problems for their reactions. If the list were presented first, respondents might choose only social problems from your list rather than reporting their own notions.

Question Content

Whatever form a questionnaire or interview takes, it's important to ask the questions in language meaningful to the respondent, especially in mailed questionnaires where you're not present to clear up confusion. Even when you're present, however, people often make assumptions about the meaning of questions, and if a respondent misinterprets something, you may not know this until after the questionnaire's been returned. (to be continued)

Limitations to Questionnaires and Interviews

It's a major limitation to questionnaires and interviews that respondents may refuse to participate. This may be particularly problematic with telephone interviews, where the reluctant respondent can simply hang up the phone, and with mailed questionnaires where a carefully prepared measuring device may be summarily tossed in the trash. Journal articles frequently report the percentage of persons who refuse to be interviewed or fail to return the questionnaire. The rate of non-response is usually at least 20 to 30 percent with mailed questionnaires and is very often larger than 50 percent. Non-response to interviews tends to be lower.

Respondents may refuse to cooperate with the researcher for a variety of reasons. In some instances the reasons may be physiological, as when a respondent is blind, deaf, or in poor health. In other cases there may be a language barrier. But most commonly respondents claim they're too busy, the research is worthless, or they have negative feelings toward the interviewer or the agency sponsoring the research. In these cases you should be prepared to convince respondents the time commitment necessary for participation is worthwhile.

The important question is how these non-responses affect the generalizability of findings. If we were surveying attitudes toward working mothers and had a large number of female non-respondents who were "too busy" or not at home, we'd suspect those who failed to respond differed significantly from those who cooperated. While this problem of selectivity is a major concern, one of several strategies may be used to assess the degree of representativeness. One method is to compare selected demographic characteristics of the sample with known characteristics from other sources. For instance, the distribution of race, income, or education from completed interviews or returned questionnaires might be compared with census data for the area in which they were gathered. A second strategy for those doing home interviews is asking neighbors some basic questions, such as number in household, occupation, and age of those refusing or not at home, then comparing the distribution of these characteristics with those of respondents who completed the interview or questionnaire. A third technique with mailed questionnaires is to compare the responses of early and late respondents. The assumption here is that the late respondents are more similar to non-respondents. If late respondents differ significantly from early respondents, generalizations to the larger population may be questionable.

Besides total non-response, there may be selective non-responses to certain questions. If questions ask about behavior or emotions from the past, respondents may simply not remember. There may also be questions respondents didn't ever possess the knowledge to answer. Respondents may have little information or not have thought about the issue that is the subject of an opinion question. Finally, individuals may refuse to answer questions simply out of a concern for privacy. In all these instances, probing for an answer may decrease non-response, but the resultant answers may also be of questionable validity.

Another problem is the possibility that respondents aren't telling the truth or are distorting their actual beliefs or facts about their lives. Would you expect narcotics peddlers or prostitutes to disclose their real occupations? In addition, there are issues related to socially undesirable norms such as prejudice or alcoholism, where the respondent may be more concerned with answering "the right way" than divulging true attitudes or behaviors. In some cases you can compare interview or questionnaire data with other sources. For example, medical records may be checked for vaccinations or police records for number of arrests. However, in many instances there's little you can do beyond being polite and non-evaluative and assuring complete confidentiality.

As you've seen, the presence of an interviewer has distinct advantages in decreasing the number of non-respondents and being available to clarify questions. At the same time, the presence of an interviewer can limit or bias a study's results. Because interviews require a great deal of time and effort, it's usually necessary to hire and train interviewers, which makes this technique for data collection the most expensive. In addition, some interviewers may "fake" parts of interviews, either because they don't want to spend the time or for some reason find interviews difficult to schedule or conduct. People hired for this work usually don't have the same degree of commitment as those sponsoring the research. For this reason it's necessary to make interviewers feel they're an important and vital part of the project, both by explaining the nature of the research and by careful training. It's helpful to make "spot checks" by re-interviewing a selected sample of the study population, but this adds to the expense and effort required.

In spite of the interviewer's attempts to place the respondent at ease, subtle cues resulting from the interviewer's characteristics—such as race, age, ethnic membership, and gender—can influence the truthfulness of respondents' answers, particularly if the variables under investigation are related to these characteristics. For example, in one study two equivalent samples of Christian respondents were interviewed, one group having a Christian interviewer and the other a Jewish interviewer. When asked if they thought Jews had "too much influence," 50 percent of those with a Christian interviewer said "Yes," while only 22 percent of those with a Jewish interviewer said "Yes." If you know interviewer characteristics might be a source of bias, one solution would be to remove them from the study. Another solution would be to measure these effects. Neither of these solutions may be very practical, because they increase the time and effort involved in planning and may increase the number of groups necessary to include in the research design. In addition, it's not always possible to determine what characteristics will be a source of bias, so probably the best way to minimize this problem is to assign interviewers randomly.

One final problem is the possibility of interviewer bias. Many of the procedures discussed earlier—such as training interviewers to be non-evaluative, to avoid leading questions or inserting their own point of view, to use tactful probing, and to make an accurate recording of responses—are attempts to eliminate this source of bias. Still, the interview situation finally depends on an interviewer reading a question to a respondent. Clearly all respondents to questionnaires at least get exactly the same question, though they may not interpret it identically. When using interviewers, it's important to let them know they may not under any circumstances edit questions or substitute wording they think is better. As the questioning proceeds, more subtle effects may occur if the interviewer begins to make assumptions about respondent attitudes and opinions that influence the way the interviewer hears and records answers. Perhaps the only way to detect such interviewer bias is to use hidden observers, obviously a difficult, deceptive, and time-consuming procedure. As with other methods of data collection, try to assess to what degree interviewer bias may exist and whether or not it could influence interpretation of findings.

OBSERVATIONAL TECHNIQUES

Given the multitude of problems associated with asking questions about behavior, direct observation might seem to offer a sounder technique for data collection than questionnaires or interviews. For some types of problems it clearly does.

Types of Observations

In the process of collecting data through observation you may be a participant or nonparticipant. This distinction depends on whether or not you take part in the behavior being studied. In either case, you can either identify yourself as an observer or disguise that purpose. For example, participant observers have become members of communities or groups and recorded observations without in any way setting themselves apart from the people being studied. This method was used by a group of sociologists who attended a Billy Graham crusade in New York as members of the audience.

In nonparticipant observation,  others may also remain unaware of your role. For example, standing on a corner to watch how many people jaywalk need not be obvious to pedestrians. In most cases, however, nonparticipant observers are identified in their role as researcher. A classic example is Goffman's observations of the "performances" in everyday interactions in a Shetland Island community, the basis for extended research supporting his Role Theory. He used the theatrical analogy of "front-stage behavior" to describe waiters and waitresses serving guests in a restaurant, contrasted to their "backstage behavior" in the kitchen where guests could not overhear their conversations. Had Goffman been a guest of the restaurant, he wouldn't have been able to observe their backstage behavior.

A great advantage of immersion into an ongoing situation is a reduction in the possibility of misinterpretations arising from ignorance. For example, different groups of people use terminology meaningful only to their particular groups. When you participate totally in their lives, you can come to understand these subtle variations in language and its context. This is illustrated by an observational study of a drug subculture and its use of the word "pinched." Had the observer allowed his assumption to go unquestioned, he would have equated a "pinch" with an "arrest." Later it became clear that a "pinch" was an arrest for heroin use in such common circumstances that details were rarely offered. Thus, through covert membership in this group, the observer came to understand a very specific meaning for being "pinched."

Another distinction in observational techniques is whether they're structured or unstructured. No observational procedure is completely unstructured, or course. The difference is one of degree. Spending time within a subculture and making non-qualitative ethnographic notes is an example of a relatively unstructured observation. On the other hand, there are techniques for observing small-group situations where exhaustive categories are used to classify all behaviors that occur. An example of a highly structured observational scheme is Bales' Interaction Process Analysis (IPA).  This scheme is an attempt to record all problem-solving activities of a group. Observers using IPA must record each unit of verbal or nonverbal  behavior using the categories indicated. At the bottom are listed the six categories Bales suggested affect group functioning: (a) problems of communication, (b) problems of evaluation, (c) problems of control, (d) problems of decision, (e) problems of tension-reduction, and (f) problems of reintegration. In addition, Bales further categorized small group interactions into social-emotional (positive or negative) and task areas. Facility in recording IPA observations requires much training and is generally done with the aid of special machines developed for this purpose. IPA observations often occur in a small group room's observation booth with a one-way mirror.

Structured observations may also take place in more natural settings. Albert Reiss, for example, developed procedures to study police interactions with the public. Very specific questions recorded police interactions when they were dispatched. Studies of parent-child interactions often use structured observation. These might include such dimensions as whether the mother issues orders to the child without explanation or whether she tries to distract the child with interesting activities.

In less structured settings data might be recorded much as you keep a diary or journal of your everyday experiences. If you're looking for new insights you may attempt to record nearly everything that takes place, which could take a large amount of time. Most researchers eventually develop a shorthand for recording types of interaction that continually repeat themselves. They're also likely to quit recording observations that appear to have little relevance to the study.

In choosing the best observation method, the first criterion is whether those being observed know about your role as observer. If your role is to be disguised, jotting down conversations in a notebook would, of course, arouse suspicion, requiring that you record observations later. Where people know they're being observed, tape recorders, obtrusive note taking, or more elaborate scoring forms may be most efficient.

Limitations to Observations

Determining the extent to which your presence affects an ongoing interaction is most difficult. Whenever people know they're being observed, they might alter their behavior, creating a source of bias. Even if you're a covert observer, your presence may change what might otherwise have occurred. The difficulty of pure observation is an ever-present problem. You should make every attempt to insure that your observations are not distorted by your own perceptions and biases.

This problem is partly one of objectivity, which is particularly problematic for the participant observer, who may become totally immersed in the situation. An example is the study of the Billy Graham crusade, where two of the observers left their group and joined others in the audience "making a decision for Christ." When the situation's more structured and you're a nonparticipant, objectivity is likely to be enhanced, but at the same time your role is more evident and may evoke unintended reactions.

Special problems may arise in sampling for an observational study. Many times observational studies are also case studies, and there are problems of generalizability and control associated with case studies (see Chapter 4, Study Design). However, these problems are not inherent in observation. The kind of sampling problem that can arise in observational studies has to do with the researcher's proximity to those being studied. If you wish to observe interactions within a subculture of drug users, you must be willing to live or interact within such settings, and may rationalize that college students who use drugs are a reasonable substitute. Remember that social scientists tend to be middle-class and find some settings alien and difficult.

Finally, the consequences of a potential loss of privacy must be considered. There's always an ethical issue involved when people are being deceived, and this problem may increase when they don't even know they're being studied. It is now federal law that an Institutional Review Board (IRB) must evaluate the potential physical or psychological risk of research involving human subjects before the study begins, including any surveys or questionnaires to be used in a project.

AVAILABLE DATA

Sources of available data are so varied and numerous that possibilities for their use are limited only by your imagination. Social research has been conducted using measures of wear and tear on museum floors to determine exhibit preferences, frequency of phone calls to detect communication patterns, records of grades to investigate influence of roommates on academic achievement, and worn pages of books as indicators of their popularity. Sex differences in erotic interests have been examined using graffiti in public toilets, and historical life expectancy has been investigated using dates of birth and death on tombstones. Available data is a great advantage over other data collection techniques because it's generally less expensive and less time-consuming than gathering new data. Sources of such data can be subsumed under two very general categories: written sources and physical evidence.

Written Sources--The Bureau of the Census publishes one of the most complete sets of available written data. While census data are gathered by questionnaires and interviews, they are available to researchers as written documents in the sense used here. Any number of other written sources can be used for social research. Many organizations, for-profit and not-for-profit, collect ongoing data of various kinds, such as performance reviews, employee or patient satisfaction surveys, and profitability measures.

Physical Evidence--You may be interested in obtaining information for which there are no written sources, or which is difficult to elicit verbally or gain by observing behavior directly. In these cases physical evidence can be used. For example, one researcher--who wanted to estimate the amount of liquor purchased in a Massachusetts city that had no package stores--counted the number of empty liquor bottles in trash cans of city residents.

Content Analysis

Content analysis is a technique to objectively, systematically, and quantitatively describe manifest communication content. Communication in this context can refer to newspapers, speeches, music, novels, essays, art, or many other sources of manifest content. The key here is that the analysis of the content of these communications must be systematic. Thus, reading a book and then reporting on the content, as in a book review, is not content analysis. Rather as with all techniques we've discussed thus far, the medium of communication must be adequately sampled, a unit of analysis chosen, and variables organized into nominal or quantitative categories to answer questions of interest.

This techniques has been used in a variety of settings. It gained widespread and popular use during World War II to analyze speeches and propaganda, when researchers hoped to gain information on troop movements or plans. Content analysis has been used to describe the content of advertisements, letters, newspaper articles, movies, songs, literature, and transcripts of small-group meetings.

It's true that when we look at a painting or read a newspaper, we have a number of impressions about content. But how can such impressions become sufficiently organized that others will agree with them? First, for content analysis to be systematic it must have a regularized sampling procedure. The sample in this case is not of individuals but of some unit of the communication. For example, if you're analyzing the lyrics of songs, the sample might be the top ten songs during the month of January for the past five years. If you're interested in comic strips, your sample might be the Sunday comic strips for the past two years. Or, if you're interested in presidential speeches, you might sample the State of the Union addresses given every fifth year for the past twenty-five years.

As in all sampling, the procedure should maximize generalization to some universe of interest by being as representative as possible. Pursuing one of these examples, we might analyze the content of hit songs for insight into the major societal themes during a given time period. If we chose only the song we personally liked, no generalizations would be possible. Likewise, we couldn't generalize from what we find if we chose what's interesting to read from the comic strips, or the best speeches.

After selecting the sample you plan to analyze, the next step is to choose units within the sample to be categorized and to design categories for that purpose. For example, suppose you choose to analyze presidential State of the Union addresses at five-year intervals over the past twenty-five years, your general purpose to document important social problems over that period of time as seen by the head of state. The units to be analyzed within these speeches might be paragraphs, categorized according to their overall meaning; they could be sentences, categorized by general topic, or they could be words, counted by frequency of occurrence. There are many other possibilities. The choice of one unit of analysis depends on the nature of the research question. If your interest in presidential speeches concerns general themes, then paragraph meaning might be most appropriate. If you're concerned with how often economic references are made, regardless of the exact content, then words or phrases might be more useful.

Categorizing these units should allow statistical manipulation--so the data can be transformed into nominal, ordinal, or interval-ratio scales. They might represent the presence, intensity, or frequency of dimensions of content. You could, for example, count the number of times "war" is mentioned by the President, either in word, phrase, sentence, or meaning. That would be a measure of this theme's frequency, and certainly of its presence. Or measuring something about intensity, you might look at the context in which war is mentioned, judging statements about arms or weapons to be more intense than statements about troop safety.

Because each problem will carry its own limitations and necessities, it's important that the categories and units you use are consistent throughout the study and make sense for the purpose outlined in the research. In this way, the communication content becomes organized into categories that will answer specific problems. More important, you've moved beyond the world of individual impression and described the communication content in a way that it could be duplicated by others.

It should be apparent that we're describing a method for both data collection and data organization. Content analysis has much to offer, because it provides a mechanism to systematically study the content of the many forms of communication that make up a substantial part of modern culture. However, because communication takes on many different forms and researchers have tried to analyze it for many diverse purposes, there's little agreement on what actually is and is not content analysis. Some have even questioned the utility of such a procedure, no matter how it's performed. These questions are largely centered around the reliability and validity of the procedure.

The reliability issue is whether, after a particular analysis is performed, it can be replicated by others. The likelihood of replication being possible is increased by careful specification of the procedures followed in the first place. While this may sound easy, the quest for exact specification of procedure in content analysis may lead to the maximization of trivia. For example, in the study of State of the Union messages, we might decide that to maximize reliability we'll merely count the number of times "war" is actually mentioned. If we also report exactly which speeches were analyzed, the methodology can be duplicated. Still, we're likely to come up with results something like the following: "While the President mentioned war six times in the first year under investigation, he mentioned it only three times the year thereafter and then seven times in the final year." The relevance of such findings is obviously limited.

Supposed we'd taken a different tack and decided to record the intensity of war mentions. From recordings of the speeches and listening to the portions on war, we may judge their intensity on some scale from say, 1 - 7. The data reported from this investigation may be more rich, but could they be duplicated? As in any reliability check, we could compute measures of speeches among our own raters, for some assurance. Still, the more complex the categorization, the more difficult becomes exact description, and thus we more we're suspicious of its reliability.

The content analyst must walk a careful line to maximize the importance of analysis while still making the methods reliable. As computer technology becomes more advanced in dealing with language symbols, some of the counting tasks common to content analysis have been assumed by machines. Computer programs can now do word searches, and this may increase reliability. Still, computers cannot deal with the semantic complexities of war unless all synonyms or combinations of words are programmed.

The validity problems of content analysis are not inherent in the method but in the purposes for which the resultant data are used. If we systematically analyze the State of the Union speeches over time, are we measuring what the audiences received, what the speakers intended, or something else? The definition above specified that the communication content to be analyzed must be "manifest." But that doesn't clear up to whom such content is manifest. The content analyst is on the least shaky ground here if procedures are carefully spelled out and no claim is made that either the intent of the original communicator or what was received by original audiences has necessarily been captured. Rather, given the sampling and categorization scheme used by the researcher, we argue that the currently perceivable, or manifest, content of the communication has been made clear, at least along the dimensions of interest in the project.

Limitations to Available Data

The lure of using available data is primarily the time and money saved by using sources already in existence. In general, then, it's an efficient strategy to make a thorough search for information that might be turned to good use in a project before you embark on extensive new data collection. There are some difficulties with available data, however.

First, because you don't control the data-gathering process, knowledge about your topic may be imperfect. Suppose you wish to study the controversy surrounding Nixon's resignation in 1974. If you choose  the Watergate Tapes as a primary source of information, you might miss the 18-1/2 minutes that were initially missing. If so, you'd have a selection problem. While this particular example has received so much publicity you'd quickly discover the infamous 18-1/2 minutes, such a discovery can't always be assured. "Secret" papers appear all the time, causing a virtual rewriting of history.

Known omissions may plague a researcher as well. For example, in spite of phenomenal efforts to correct the problem, the United States population is typically undercounted by the U.S. Census. The Census Bureau estimates that the 2000 census, for example, missed 6.4 million people. Worse than that, the undercount is selective -- mostly minorities, children, and low-income people are those not counted. The Census Bureau also estimates that 3.1 million were counted twice in 2000 -- mostly white and affluent people; who probably received two census forms to fill out because they own two homes. Even if the census were complete, such important social variables as religious affiliation are not included.

The form of the information available may also present problems. Perhaps you plan to use a public opinion poll that includes both religion and occupation, but categorizes occupations into only six broad groups. To answer your questions of interest, you may require more detail.

Such problems challenge you to decide whether the project and theoretical development of your topic justify the expense and effort of collecting original data. For some topics, of course, there's simply no information already available.

PILOT STUDIES AND PRETESTS

Whether a study is to employ questionnaires, interviews, observations, or available data, a pretest is vital. Pretests are preliminary tests of the measures to be used on a small sample of the population to be studied. No matter how carefully you design a measure, it's still wise to give it an actual try. A pretest of a questionnaire may demonstrate that some of the questions are unintelligible to respondents. In a questionnaire with open-ended questions you might find that respondents aren't giving adequate answers, suggesting a need to reword the questions. You may find in a pretest that respondents don't feel the interview is legitimate and refuse to be questioned, signaling the need to reword the introductory remarks. If you're planning to observe a small group you may find your method of recording isn't exhaustive enough to categorize all the behaviors being measured.

Even with available data, proposed methods may not work owing to some unexpected characteristic of the data. For example, in planning to use available police records for a study of delinquency, you may find in the pretest that juvenile records are much less complete in some precincts than in others.

A pilot study is more comprehensive than a pretest, in that it involves carrying out the entire research project in miniature, including selecting a sample, pretesting measures, training assistants, gathering data, assessing sources of bias, and coding and analyzing the data gathered. These procedures can provide valuable information and save you a great deal of time and money by helping eliminate problems from the research design before the final project is carried out. In addition, preliminary tests of hypotheses are possible. If the pilot study were to demonstrate that proposed relationships between variables do not seem to exist, you may be alerted that a large-scale study is not warranted.

Pilot studies and pretests are particularly essential when investigating topics about which little is known or when employing unprecedented measures. Indeed, information generated from a pilot study is sometimes so interesting either in results or methodological procedures that they justify publication.

ISSUES IN DATA COLLECTION

Reactivity

The act of studying the world changes the world. If you were to measure this book with a ruler, you'd dislodge several atoms, although the difference would not be enough to be noticed in future measurements. More extremely, some tests for assessing the amount of an element in an ore sample may change the original sample beyond recognition. In a similar manner, social scientists are concerned their methods of measuring social phenomena might alter those phenomena. For instance, having taken IQ or similar tests increases future scores on those tests. It's important, then, to keep in mind that measurement is not passive like listening to the radio, but more like a football game where the spectators themselves may influence the outcome of the game. Your ability to deal adequately with this characteristic of any data collection procedure is crucial for understanding the results of a particular study effort.

Testing Effect

When respondents know they're being observed or studied, that is, when your measures are obtrusive, it's always possible they may react to this awareness with responses you don't anticipate. The classic example of this effect is a study carried out at the Hawthorne Electrical Company in the 1930s. The investigators asked if various improvements in working conditions would lead to an increase in productivity of workers making electrical equipment. A group of workers was placed in a separate room and their productivity measured after such changes as better lighting, longer rest periods, or greater financial incentives. Each change in working conditions was associated with an increase in productivity. Had the study terminated at that point, the conclusion might have been that these changes alone were the cause of increased productivity. However as a check on their results, the investigators returned to the original conditions. To everyone's surprise the workers' productivity still increased! Clearly, some extraneous variable was having an effect on productivity, and alternative hypotheses had to be considered. After questioning the workers, the investigators argued that these subjects were highly motivated to work harder because the extra attention they'd been given made them feel their work was important. This particular study has been cited so often, respondents' responses that are effects of the test itself are sometimes called the "Hawthorne Effect."

Demand Effect

Another type of reactive effect occurs when participants feel certain responses are being "demanded." Having little or no information about the hypotheses being tested, they try to figure out the "real" purpose of the study, and act in ways they think the investigator expects. To examine this effect, subjects were instructed to add sums of numbers and to read an instruction card after completing each page. In every instance the card instructed them to tear up the work just completed and continue with the next sheet of paper. The experimenters expected the subjects to stop when they realized all the instruction cards were the same, but in fact they continued for hours. When interviewed later, they said they viewed the task as part of a legitimate experiment, perhaps having to do with some sort of "endurance test." In other words, they decided endurance was what the experimenter demanded of them, so tried to cooperate.

This kind of effect can occur in questionnaire or interview situations when respondents supply answers they perceive the interviewer wants.

A demand effect can occur during apparently unobtrusive observations as well. For example, if you were periodically checking police records to test a hypothesis about juvenile delinquency rates across various precincts, the clerical personnel may unknowingly make systematic changes in the records simply because the files are under scrutiny. Perhaps the records hadn't been very accurate when the investigation started and the record keeper is attempting to increase accuracy; or perhaps a person in one of the precincts wants to make that area appear less crime-prone.

One way to detect such sources of bias is a post-data-collection interview. By asking people if they were aware of deception and by careful probing, it's sometimes possible to assess the amount of bias. Still, there's always the possibility subjects may be unaware they've been affected by the measurement procedure.

Role Selection

While most of us see ourselves as having stable and consistent personalities, we still take on different roles in different settings. For example, you may behave quite differently in the presence of your boss or instructor than you do with your spouse or best friend. Role selection occurs in data collection when subjects or respondents feel they are under certain role expectations. For example, even if told they're part of a random sample, they may also be told their opinions or responses are special and important to the study. Under such instructions they may feel the necessity to assume the role of "expert" relative to the subject being studied. If there are questions to which they don't know the answer, they may tend to supply some answer in keeping with the "expert" role rather than choosing a "don't know" or "no opinion" response.

Each of these examples of reactive effects illustrates ways the very phenomenon about which we're gathering data may be changed by our attempts to capture it. The interview study is subject to bias introduced by the interviewer's characteristics, the setting where the interviewing takes place, the questions, and many other factors. Observational studies suffer from the apprehension of subjects when they know they're being stared at from behind a mirror or recorded by the scientist-stranger. Even the use of available data, if announced to those who control such data, may include systematic bias.

Investigator Bias

It should be clear at this point that any number of potentially biasing effects may threaten the validity of social research. There are at least two parties to any research endeavor, however, and you shouldn't hasten to blame bias entirely on the people being studied. It's been amply demonstrated that you can introduce substantial bias as the investigator. Your bias may be the subtle result of hoping to find support for a favored hypothesis, or it may be the outcome of your limited information about the subjects or the phenomenon under study. As a corollary to the demand effect in the people you're studying, you may unintentionally convey your expectations. This problem is particularly great because people are so susceptible to an investigator's influence.

Investigator bias is exemplified by a series of studies where student "experimenters" were asked to run groups of twenty in a "person perception task." The people being studied were to rate photographs of faces as showing evidence of success or failure. The "experimenters" were given identical instructions, except that half were told previous research had demonstrated subjects would judge the faces as evidencing failure; the other half were told the faces were usually judged as relatively successful. Even though the "experimenters" knew the photos showed neutral faces, and even though they simply read the instructions and had no further contact with the people being studied, the groups assigned to "experimenters" who expected ratings of success did, in fact, support this expectation; for groups assigned to those who expected ratings of failure, the ratings met "failure" expectations. The experimenter effect was somewhat reduced when repeated without visual cues, but there was still some effect, even when taped instructions were used.

An even more dramatic example of this effect is a study of maze-learning with mice. One set of research assistants, acting as investigators, were told their group of mice were exceptionally fast learners and could learn to run mazes quickly. Other students were told their mice were slow learners, although the mice were actually very much the same in their maze-learning capacities. The time allowed and number of trials were the same for each group of mice. As you can anticipate by now, the mice taught to run mazes by graduate students who thought the mice were exceptionally able indeed showed superior performance to mice run by students who assumed them to slow learners. In some way, these expectations were communicated to the mice and affected their performance.

Investigator bias can be reduced by more careful selection and training of investigators, minimizing contact with subjects, careful checking of data for computation errors, increasing the number of investigators, using taped or written instructions, and observing the investigators for biasing behaviors. In addition, purposes or hypotheses of research may be concealed from those collecting the data. An interesting technique for accomplishing this, and minimizing respondent bias as well, is the double-blind procedure. Using this technique, neither the subjects in a research project nor those taking measures know the hypotheses of the research or the characteristics of those they're measuring. For example, a medical researcher may be interested in the side effects of a given drug. Some patients are given the drug while others are given a placebo. Patients may be asked to have physical examinations and to report on their physical symptoms at regular intervals. Using a double-blind procedure, neither patients nor those examining them know whether they were given the real drug or the placebo. In experimental terminology, this means neither those doing the exams nor the patients themselves know whether they're in the experimental or control groups. This precaution has obvious advantages--minimizing the temptation for subjects to exhibit drug side effects and minimizing researchers' over-zealousness in finding these side effects (or not finding them).

Approximations to Social Reality

In the discussion of questionnaires and interviews we pointed out that the vast majority of social science research employs only these two data collection strategies. This has led some researchers to argue that social science has become the science of verbal behavior, relying on reports of behavior rather than direct measurement of the behavior itself. Because each social interaction is a unique and complex event, self-reports don't necessarily predict how a person will behave. In one study, the investigator visited 251 auto camps, hotels, and restaurants with a Chinese couple and all but one of these establishments accepted them as guests. Six months later the same establishments were sent a questionnaire that included a question asking if they'd accept Chinese guests, and all but one said they would not.

The above sections on limitations to questionnaires and interviews and of respondent and investigator bias must have made clear the many potential sources of contamination for data gathered by these strategies. If we ask respondents for self-reports about certain child-rearing practices, how do we know we've obtained information about their actual behavior toward children? Can we study such a topic without actually watching the behavior in question? It could be that the proper statement of results from such a study indicates the reported use of certain child-rearing practices. Of course, the more embarrassing or non-normative the behavior we're inquiring about, the more difficult it becomes to be certain we've measured that behavior by verbal reports.

To some degree we should expect discrepancies between verbal and actual behavior. Common-sense phrases based on everyday experiences of such discrepancies are plentiful; for example, "Put up or shut up," and "Talk is cheap." A compromise solution to this dilemma is not to abandon questionnaires and interviews but to consider honestly whether or not direct observation of the phenomenon of interest might also be possible.

Considering this alternative, however, the resolution of this problem is less clear. If we limit social science to the investigation of only those behaviors we can actually observe, we add a multitude of other problems. How are we going to get parents to let us move into their homes and watch them discipline their children? By the time enough parents in enough homes have been observed to generalize results adequately, the time and dollar expense would be enormous.

We can at least provide systematic, as opposed to haphazard, accounting of the biasing effects of questionnaires and interviews. Such accounts can only be made as we store data on various factors that affect response. When this information becomes more complete (and a great deal is already available), then characteristics of the interview or questionnaire situation may be included as controls in sorting out the real relationships between variables being studied.

This problem is really one of validity. On the one hand, we're well aware of the potential biasing effects with questionnaires, interviews, and available data. On the other hand, observation as a method of data collection also has its sources of bias. Except in the most limited cases where opinion might be unanimous, social scientists are left with a vast array of data for which there will never be 100 percent agreement among observers; even if there were, the question of validity seems ultimately unanswerable, except within the realm of our interpretations as fallible and biased human beings. This is one reason we've suggested it's often important to use more than one source of data in attempts to abstract meaning from social variables.

EXERCISES

1. Practice writing a cover letter for a questionnaire. After you compose your letter, check your work against the criteria discussed in this chapter. Would your letter encourage people to respond?

2. Go to the library or search on the internet for a research monograph that includes the questionnaire used in the research. Fill out the questionnaire yourself, so you'll have some idea of how respondents might have reacted. Then critique the instrument, using the guidelines presented here.

3. Critique the following excerpts from a questionnaire designed for new trainers, rewriting where necessary. Clarify why you suggest your changes:

   Circle the numbers of the items most characteristic of participants or other aspects of the workshop you teach.

   1. There is no major problems but many little unrelated minor problems. Or there are no major problems related to the list here but there are other major or minor problems.

   2. The material is clearly organized and easy to learn and it seems foolish to lecture on all of it.

   3. Many participants could move more rapidly through the workshop and this creates boredom for them.

4. Demonstrate that you understand what a leading question is by writing one and then correcting it.

5. Practice probing techniques with someone else, one of you playing the respondent and the other playing the interviewer. Begin with a question like "Why did you take this course?" Probes might include such phrases as "Any other reason?" or "Is that all you can recall?" Were your probes working? How did the respondent feel?

6. Select a group meeting where you can be a participant observer. Define a single dimension to observe, such as the number of comments made by each member. Describe why the dimension is important to observe. Devise a scheme to record your observations, and make a report.

7. Find out what kinds of research your closest university keeps that are available to the public. What research questions could be answered using these records? Write a short proposal for their use.

8. Based on the latest census information, write a demographic and social profile of your home community. Include such information as family income or housing characteristics. What other information would you have liked to find?

9. Find two data-based published journals that interest you. Which of the issues in data collection presented in this chapter were discussed in these studies? How many data collection problems did the authors discuss? Critique the data collection procedures used by the authors.

Back to Social Research Preface and Index