Hide Notes 

Slide 2: Asking Questions The Definitive Guide to Questionnaire Design— For Market Research, Political Polls, and Social and Health Questionnaires, Revised Edition
Slide 5: Asking Questions
Slide 6: Norman M. Bradburn Seymour Sudman Brian Wansink
Slide 7: Asking Questions The Definitive Guide to Questionnaire Design— For Market Research, Political Polls, and Social and Health Questionnaires, Revised Edition
Slide 8: Copyright © 2004 by John Wiley & Sons, Inc. All rights reserved. Published by Jossey-Bass A Wiley Imprint 989 Market Street, San Francisco, CA 94103-1741 www.josseybass.com No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, e-mail: permcoordinator@wiley.com. Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002. Jossey-Bass also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Bradburn, Norman M. Asking questions : the definitive guide to questionnaire design—for market research, political polls, and social and health questionnaires / Norman M. Bradburn, Brian Wansink, Seymour Sudman.—Rev. ed. p. cm. Earlier ed. by Sudman and Bradburn with Sudman named first. Includes bibliographical references (p. ) and index. ISBN 0-7879-7088-3 (alk. paper) 1. Social sciences—Research. 2. Questionnaires. I. Wansink, Brian. II. Sudman, Seymour. III. Title. H62.B63 2004 300'.72'3—dc22 2004001683 Printed in the United States of America FIRST EDITION PB Printing 10 9 8 7 6 5 4 3 2 1
Slide 9: Contents Preface and Acknowledgments The Authors Part I. Strategies for Asking Questions 1. The Social Context of Question Asking Part II. Tactics for Asking Questions 2. 3. 4. 5. 6. 7. 8. 9. Asking Nonthreatening Questions About Behavior Asking Threatening Questions About Behavior Asking Questions About Attitudes and Behavioral Intentions Asking and Recording Open-Ended and Closed-Ended Questions Asking Questions that Measure Knowledge Asking Questions that Evaluate Performance Asking Psychographic Questions Asking Standard Demographic Questions Part III. Drafting and Crafting the Questionnaire 10. 11. Organizing and Designing Questionnaires Questionnaires from Start to Finish xi xv 3 35 79 117 151 179 213 243 261 283 315 vii
Slide 10: viii CONTENTS 12. Asking Questions FAQs 323 335 347 367 375 411 417 425 Bibliography and Recommended Readings Glossary Index Appendix A: List of Academic and Not-for-Profit Survey Research Organizations Appendix B: Illinois Liquor Control Commission: College Student Survey Appendix C: Faculty Retention Survey Appendix D: Kinko’s: Open-ended Service Satisfaction Survey
Slide 11: This book is dedicated to the memory of our colleague and coauthor Seymour Sudman who died tragically while we were in the midst of writing this book. His spirit and wisdom have continued to inspire us as we brought this manuscript to press. He lives on in this book.
Slide 13: Preface This book is a revised and updated edition of Asking Questions: A Practical Guide to Questionnaire Design, first published in 1982. It focuses on the type of question asking that social science researchers and market researchers use in structured questionnaires or interviews. Many of the principles of effective formalized questioning we focus on in this book are useful in other contexts. They are useful in informal or semistructured interviews, in administering printed questionnaires in testing rooms, and in experimental studies involving participant evaluations or responses. We intend this book to be a useful “handbook” for sociologists, psychologists, political scientists, evaluation researchers, social workers, sensory scientists, marketing and advertising researchers, and for many others who have occasion to obtain systematic information from clients, customers, or employees. In the past two decades, two major changes in the practice of survey research prompted us to produce a revised edition. First, there has been a revolution in research on question asking brought about by the application of cognitive psychology to the study of questionnaire design. We now have a conceptual framework for understanding the question-answering process and the causes of the various response effects that have been observed since the early days of social scientific surveys. This work has helped move questionnaire construction from an art to a science. Second, there has been a technological revolution in the way computers can be used to support the survey process. Computerassisted survey information collection (CASIC) refers to a variety xi
Slide 14: xii PREFACE of specialized programs used to support survey data collection—for example, CAPI (computer-assisted personal interviewing) or CATI (computer-assisted telephone interviewing), to name the most common forms of CASIC. The greater use of computer technology at every stage of data collection in surveys has made many of the suggestions in our earlier edition obsolete and necessitated a thorough reworking of discussion that was predicated on traditional paper-and-pencil questionnaires. We are also beginning an era of Web-based surveys. Although there is still much to learn about this new method of conducting surveys, we have tried to incorporate what we know at this time into our discussions where relevant. We have tried to make the book self-contained by including major references. Some readers, however, may wish to refer to our earlier books, Response Effects in Surveys: A Review and Synthesis (Sudman and Bradburn, 1974); Improving Interview Method and Questionnaire Design: Response Effects to Threatening Questions in Survey Research (Bradburn, Sudman, and Associates, 1979); Thinking About Answers (Sudman, Bradburn, and Schwarz, 1996); and Consumer Panels, (Sudman and Wansink, 2002), for more detailed discussion of the empirical data that support our recommendations. This book is specifically concerned with questionnaire construction—not with all aspects of survey design and administration. Although we stress the careful formulation of the research problems before a questionnaire is designed, we do not tell you how to select and formulate important research problems. To do so requires a solid knowledge of your field—knowledge obtained through study and review of earlier research, as well as hard thinking and creativity. Once the research problem is formulated, however, this book can help you ask the right questions. The book is divided into three parts. In Part I we discuss the social context of question asking. We present our central thesis, namely that questions must be precisely worded if responses to a survey are to be accurate; we outline a conceptual framework for understanding the survey interview and present examples to illus-
Slide 15: PREFACE xiii trate some of the subtleties of language and contexts that can cause problems. We also discuss some of the ethical principles important to survey researchers—the right to privacy, informed consent, and confidentiality of data. Part II is devoted to tactics for asking questions. In Chapters Two through Nine we consider the major issues in formulating questions on different topics, such as the differences between requirements for questions about behavior and for questions about attitudes. We also consider how to ask questions dealing with knowledge and special issues in designing questions that evaluate performance, measure subjective characteristics, and measure demographic characteristics. In Part III we turn from the discussion of the formulation of questions about specific kinds of topics to issues involved in crafting the entire questionnaire. We discuss how to organize a questionnaire and the special requirements of different modes of data collection, such as personal interviewing, telephone interviewing, self-administration, and electronic surveying. We end with a set of frequently asked questions and our answers. Throughout the book we use terms that are well understood by survey research specialists but that may be new to some of our readers. We have therefore provided a glossary of commonly used survey research terms. Many of the terms found in the Glossary are also discussed more fully in the text. In addition, we have included a list of academic and not-for-profit survey research organizations in Appendix A. The chapters in Part II are introduced with a checklist of items to consider. The checklists are intended as initial guides to the major points made and as subsequent references for points to keep in mind during the actual preparation of a questionnaire. Readers new to designing surveys should read sequentially from beginning to end. Experienced researchers and those with specific questionnaire issues will turn to appropriate chapters as needed. All readers should find our detailed index of use.
Slide 16: xiv PREFACE In this book we have distilled a vast amount of methodological research on question asking to give practical advice informed by many years of experience in a wide variety of survey research areas. But much is still not known. We caution readers seeking advice on how to write the perfect questionnaire that perfection cannot be guaranteed. For readers who wish to do additional research in questionnaire design, much interesting work remains to be done. Acknowledgments While we were in the process of writing this new edition, Seymour Sudman died tragically. His vast knowledge of the research literature, deep experience, and wise judgment continue to enrich this volume. We miss him greatly. This edition builds on its predecessor and all those who contributed to it. We are indebted to many colleagues at the Survey Research Laboratory (SRL), University of Illinois, and at the National Opinion Research Center (NORC), University of Chicago. These colleagues include Herbert Jackson, who compiled the material for Chapter Twelve, and Matthew Cheney, Sarah Jo Brenner, and Martin Kator, who helped in manuscript preparation by compiling and summarizing recently published findings in the area of survey design. At Jossey-Bass, Seth Schwartz and Justin Frahm: We are grateful for their patience with the sometimes distracted authors and for their inventive solutions to the inevitable challenges that arose in turning a manuscript into an aesthetically pleasing book. Readers, as do we, owe them all a deep debt of gratitude. Norman Bradburn Arlington, Virginia Brian Wansink Urbana, Illinois August 2003
Slide 17: The Authors Norman M. Bradburn (Ph.D. Harvard University, 1960) is the Margaret and Tiffany Blake Distinguished Service Professor emeritus in the Department of Psychology and the Harris Graduate School of Public Policy Studies at the University of Chicago. He has written widely, often with Seymour Sudman, on topics in survey methodology. He was a pioneer in the application of cognitive psychology to the design of survey questionnaires. For a number of years, he was the director of the National Opinion Research Center at the University of Chicago. He is currently the assistant director for social, behavioral, and economic sciences at the National Science Foundation. Seymour Sudman (Ph.D. University of Chicago, 1962) was the Walter H. Stellner Distinguished Professor of Marketing at the University of Illinois (Urbana-Champaign) from 1968 until his death in 2000. Through a lifetime of active research, he contributed immeasurably to the area of survey design, sampling, and methodology. He was actively involved in providing guidance to the U.S. Census Bureau, and he served as deputy director and research professor of the Survey Research Laboratory at the University of Illinois. Brian Wansink (Ph.D. Stanford University, 1990) is the Julian Simon Research Scholar and professor of marketing, of nutritional science, of advertising, and of agricultural and consumer economics at the xv
Slide 18: xvi THE AUTHORS University of Illinois (Urbana-Champaign) and is an adjunct research professor at Cornell University and at Wageningen University in the Netherlands. He directs the Food and Brand Lab, which focuses on psychology related to food choice and consumption (www.FoodPsychology. com). Prior to moving to Illinois, he was a marketing professor at Dartmouth College and at the Wharton School at the University of Pennsylvania. He coauthored Consumer Panels with Seymour Sudman.
Slide 19: Asking Questions
Slide 21: Part One Strategies for Asking Questions
Slide 23: Chapter One The Social Context of Question Asking The precise wording of questions plays a vital role in determining the answers given by respondents. This fact is not appreciated as fully as it should be, even in ordinary conversation. For example, a colleague mentioned that he needed to pick out granite for a kitchen countertop. The only day he could make the trip was the Saturday before Labor Day. Although he called on Friday to make certain the store was open, he arrived at the store on Saturday only to find a sign on the door that said “Closed Labor Day Weekend.” When asked if he remembered what question he had asked the clerk at the store, he said, “I asked him what hours he was open on Saturday, and he replied ‘Nine to five.’” This story illustrates the basic challenge for those who engage in the business of asking questions. It illustrates not only the importance of the golden rule for asking questions—Ask what you want to know, not something else—but also, more important, the ambiguities of language and the powerful force of context in interpreting the meaning of questions and answers. Our colleague had unwittingly asked a perfectly ambiguous question. Did the question refer to Saturdays in general or the next Saturday specifically? The clerk obviously interpreted the question as referring to Saturdays in general. Our colleague meant the next Saturday and did not think his question could mean anything else until he arrived at the store and found it closed. In everyday life, these types of miscommunications happen all the time. Most of the time they are corrected by further conversation or by direct questions that clarify their meaning. Sometimes 3
Slide 24: 4 ASKING QUESTIONS they only get corrected when expected behavior does not occur, as was the case when the store turned out to be closed. But the stylized form of question asking used in surveys does not often provide feedback about ambiguities or miscommunications. We must depend on pretesting to weed out ambiguities and to help reformulate questions as clearly as possible—to ask about what we want to know, not something else. The thesis of this book is that question wording is a crucial element in surveys. The importance of the precise ordering of words in a question can be illustrated by another example. Two priests, a Dominican and a Jesuit, are discussing whether it is a sin to smoke and pray at the same time. After failing to reach a conclusion, each goes off to consult his respective superior. The next week they meet again. The Dominican says, “Well, what did your superior say?” The Jesuit responds, “He said it was all right.” “That’s funny,” the Dominican replies. “My superior said it was a sin.” The Jesuit says, “What did you ask him?” The Dominican replies, “I asked him if it was all right to smoke while praying.” “Oh,” says the Jesuit. “I asked my superior if it was all right to pray while smoking.” Small Wording Changes that Made Big Differences The fact that seemingly small changes in wording can cause large differences in responses has been well known to survey practitioners since the early days of surveys. Yet, typically, formulating the questionnaire is thought to be the easiest part of survey research and often receives too little effort. Because no codified rules for question asking exist, it might appear that few, if any, basic principles exist to differentiate good from bad questions. We believe, however, that many such principles do exist. This book provides principles that
Slide 25: THE SOCIAL CONTEXT OF QUESTION ASKING 5 novices and experienced practitioners can use to ask better questions. In addition, throughout the book we present examples of both good and bad questions to illustrate that question wording and the question’s social context make a difference. Loaded Words Produce Loaded Results Suppose a person wanted to know whether workers believed they were fairly compensated for their work. Asking “Are you fairly compensated for your work?” is likely to elicit a very different answer than asking “Does your employer or his representative resort to trickery in order to defraud you of part of your earnings?” One would not be surprised to find that an advocate for improving the situation of workers asked the second question. Clearly the uses of words like “trickery” and “defraud” signal that the author of the question does not have a high opinion of employers. Indeed, this was a question asked by Karl Marx on an early survey of workers. Questionnaires from lobbying groups are often perceived to be biased. A questionnaire received by one of the authors contained the following question: “The so-called ‘targeted tax cuts’ are a maze of special interest credits for narrow, favored groups. Experts agree the complex, loophole-ridden tax code makes it easy for Big Government liberals to raise taxes without the people even realizing it. Do you feel a simpler tax system—such as a single flat rate or a national sales tax with no income tax—would make it easier for you to tell when politicians try to raise your taxes?” Even an inexperienced researcher can see that this question is heavily loaded with emotionally charged words, such as “so-called,” “loophole-ridden,” and “Big Government liberal.” The authors of this questionnaire are clearly interested in obtaining responses that support their position. Although the example here is extreme, it does illustrate how a questionnaire writer can consciously or unconsciously word a question to obtain a desired answer. Perhaps not surprisingly, the questionnaire was accompanied by a request for a contribution to help defray the cost of compiling and publicizing
Slide 26: 6 ASKING QUESTIONS the survey. Surveys of this type, sometimes called frugging (fundraising under the guise) surveys, are often primarily intended to raise funds rather than to collect survey information. The American Association for Public Opinion Research has labeled fundraising surveys deceptive and unethical, but they are unfortunately not illegal. Wording questions to obtain a desired answer is not the only type of problem that besets survey authors. Sometimes questions are simply complex and difficult to understand. Consider this example from a British Royal Commission appointed to study problems of population (cited in Moser and Kalton, 1972): “Has it happened to you that over a long period of time, when you neither practiced abstinence, nor used birth control, you did not conceive?” This question is very difficult to understand, and it is not clear what the investigators were trying to find out. The Nuances of Politically Charged Issues Yet even when there are no deliberate efforts to bias the question, it is often difficult to write good questions because the words to describe the phenomenon being studied may be politically charged. The terms used to describe the area of concern may be so politically sensitive that using different terms changes the response percentages considerably. A question asking about welfare and assistance to the poor from the 1998 General Social Survey (Davis, Smith, and Marsden, 2000) produced quite different opinions. We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I’d like you to tell me whether you think we’re spending too much money on it, too little money, or about the right amount. Are we spending too much money, too little money or about the right amount on . . .
Slide 27: THE SOCIAL CONTEXT OF QUESTION ASKING “Welfare” (N = 1,317) 7 “Assistance to the Poor” (N = 1,390) Too little About right Too much Total 17% 38% 45% 100% 62% 26% 12% 100% Not all wording changes cause changes in response distributions. For example, even though two old examples of questions about government responsibility to the unemployed were worded differently, 69 percent of respondents answered “yes.” Perhaps this is because the questions were fairly general. One question, from a June 1939 Roper survey, asked, “Do you think our government should or should not provide for all people who have no other means of subsistence?” (Hastings and Southwick, 1974, p. 118). A differently worded question, this one from a Gallup poll of January 1938, asked, “Do you think it is the government’s responsibility to pay the living expenses of needy people who are out of work?” (Gallup, 1972, p. 26). Respondents are less likely to agree as questions become more specific, as illustrated by three Gallup questions from May to June 1945: Do you think the government should give money to workers who are unemployed for a limited length of time until they can find another job? (Yes 63%) It has been proposed that unemployed workers with dependents be given up to $25 per week by the government for as many as 26 weeks during one year while they are out of work and looking for a job. Do you favor or oppose this plan? (Favor 46%) Would you be willing to pay higher taxes to give people up to $25 a week for 26 weeks if they fail to find satisfactory jobs? (Yes 34%)
Slide 28: 8 ASKING QUESTIONS Note that introducing more details—such as specifying actual dollars, specifying the length of the support, and reminding respondents that unemployment benefits might have to be paid for with increased taxes—changed the meaning of the question and produced a corresponding change in responses. In later chapters we will discuss in more detail how wording affects responses, and we will make specific recommendations for constructing better questionnaires. Questioning as a Social Process A survey interview and an ordinary social conversation have many similarities. Indeed, Bingham and Moore (1959) defined the research interview as a “conversation with a purpose.” The opportunity to meet and talk with a variety of people appears to be a key attraction for many professional interviewers. By the same token, a key attraction for many respondents appears to be the opportunity to talk about a number of topics with a sympathetic listener. We do not know a great deal about the precise motivations of people who participate in surveys, but the tenor of the evidence suggests that most people enjoy the experience. Those who refuse to participate do not refuse because they have already participated in too many surveys and are tired; characteristically, they are people who do not like surveys at all and consistently refuse to participate in them or have experienced bad surveys. Viewing Respondents as Volunteer Conversationalists Unlike witnesses in court, survey respondents are under no compulsion to answer our questions. They must be persuaded to participate in the interview, and their interest (or at least patience) must be maintained throughout. If questions are demeaning, embarrassing, or upsetting, respondents may terminate the interview or falsify their answers. Unlike the job applicant or the patient answering a doctor’s questions, respondents have nothing tangible to gain
Slide 29: THE SOCIAL CONTEXT OF QUESTION ASKING 9 from the interview. Their only reward is some measure of psychic gratification—such as the opportunity to state their opinions or relate their experiences to a sympathetic and nonjudgmental listener, the chance to contribute to public or scientific knowledge, or even the positive feeling that they have helped the interviewer. The willingness of the public to participate in surveys has been declining in recent years for many reasons, one of which is the tremendous number of poor and misleading surveys that are conducted. It is therefore doubly important for the survey researcher to make sure that the questionnaire is of the highest quality. Although the survey process has similarities to conversations, it differs from them in several respects: (1) a survey is a transaction between two people who are bound by special norms; (2) the interviewer offers no judgment of the respondents’ replies and must keep them in strict confidence; (3) respondents have an equivalent obligation to answer each question truthfully and thoughtfully; and (4) in the survey it is difficult to ignore an inconvenient question or give an irrelevant answer. The well-trained interviewer will repeat the question or probe the ambiguous or irrelevant response to obtain a proper answer. Although survey respondents may have trouble changing the subject, they can refuse to answer any individual question or break off the interview. The ability of the interviewer to make contact with the respondent and to secure cooperation is undoubtedly important in obtaining the interview. In addition, the questionnaire plays a major role in making the experience enjoyable and in motivating the respondent to answer the questions. A bad questionnaire, like an awkward conversation, can turn an initially pleasant situation into a boring or frustrating experience. Above and beyond concern for the best phrasing of the particular questions, you—the questionnaire designer—must consider the questionnaire as a whole and its impact on the interviewing experience. With topics that are not intrinsically interesting to respondents, you should take particular care to see that at least some parts of the interview will be interesting to them.
Slide 30: 10 ASKING QUESTIONS Why Some Sensitive Topics Aren’t Sensitive Beginning survey researchers often worry about asking questions on topics that may be threatening or embarrassing to respondents. For many years, survey researchers believed that their interviews could include only socially acceptable questions. In the 1940s it was only with great trepidation that the Gallup poll asked a national sample of respondents whether any member of their family suffered from cancer. Today surveys include questions about a whole host of formerly taboo subjects, such as religious beliefs, income and spending behavior, personal health, drug and alcohol use, and sexual and criminal behavior. Popular commentators and those not familiar with survey research sometimes note that they would not tell their best friends some of the things that surveys ask about, such as sexual behavior or finances. The fact that the interviewer is a stranger and not a friend is part of the special nature of the situation. People will disclose information to strangers that they would not tell their best friends precisely because they will never see the stranger again and because their name will not be associated with the information. When you tell a friend about your potentially embarrassing behavior or intimate details about your life, you may worry about the repercussions. For example, Roger Brown, a well-known social psychologist, noted in the introduction to his autobiographical memoir that he deliberately did not have his longtime secretary type the manuscript of the book, although she had typed all his other manuscripts, because he did not want her to be shocked or distressed by the revelations about his personal life. He preferred to have the typing done by someone who did not have a personal connection with him (Brown, 1996). With proper motivation and under assurances of confidentiality, people will willingly divulge private information in a survey interview. Most respondents participate voluntarily in surveys. They will wish to perform their roles properly, that is, to give the best information they can. It is your responsibility to reinforce respondents’
Slide 31: THE SOCIAL CONTEXT OF QUESTION ASKING 11 good intentions by designing the questionnaire effectively. If the questionnaire requires respondents to recall past events, the question should give them as many aids as possible to achieve accurate recall. (Techniques for designing the recall type of question are discussed in Chapter Two.) Dealing with the Social Desirability Bias In general, although respondents are motivated to be “good respondents” and to provide the information that is asked for, they are also motivated to be “good people.” That is, they will try to represent themselves to the interviewer in a way that reflects well on them. Social desirability bias is a significant problem in survey research. This is especially the case when the questions deal with either socially desirable or socially undesirable behavior or attitudes. If respondents have acted in ways or have attitudes that they feel are not the socially desirable ones, they are placed in a dilemma. They want to report accurately as good respondents. At the same time, they want to appear to be good people in the eyes of the interviewer. Techniques for helping respondents resolve this dilemma on the side of being good respondents include interviewer training in methods of establishing rapport with the respondent, putting respondents at their ease, and appearing to be nonjudgmental. (Question-wording techniques that can help reduce social desirability bias are discussed in Chapter Three.) Viewing the interview as a special case of ordinary social interaction helps us better understand the sources of error in the questioning process. Conversations are structured by a set of assumptions that help the participants understand each other without having to explain everything that is meant. These assumptions have been systematically described by Paul Grice (1975), a philosopher of language. (See Sudman, Bradburn, and Schwarz, 1996, chap. 3 for a full discussion.) According to Grice’s analysis, conversations are cooperative in nature and are governed by a set of four maxims that each participant implicitly understands and shares. The maxim of
Slide 32: 12 ASKING QUESTIONS quality says that speakers will not say anything they know to be false. The maxim of relation indicates that speakers will say things that are relevant to the topic of the ongoing conversation. The maxim of quantity enjoins speakers to make what they say as informative as possible and not to be repetitive. The maxim of manner requires speakers to be clear rather than ambiguous or obscure. If the questionnaire makes it difficult for respondents to follow these maxims, an uncomfortable interaction between the interviewer and respondent can result. Respondents’ answers can also be distorted. (The importance of these principles for questionnaire design is discussed in Chapters Four and Five.) Investigators should try to avoid asking respondents for information they do not have. If such questions must be asked, the interviewer should make it clear that it is acceptable for the respondent not to know. (Particular problems relating to knowledge questions are discussed in Chapter Six.) The standard face-to-face interview is clearly a social interaction. The self-administered mailed questionnaire or those conducted electronically via the Web are much less of a social encounter, although they are not entirely impersonal. Personal interviews conducted by telephone provide less social interaction than a face-toface interview but more than a self-administered questionnaire. To compensate for the lack of interaction, the self-administered questionnaire, whether paper-and-pencil or electronic, must depend entirely on the questions and written instructions to elicit accurate responses and motivate the respondent to participate in the study. The interviewer does not have the opportunity to encourage or clarify, as would be possible in a face-to-face interview and to some extent in a telephone interview. (Differences among these modes of asking questions are discussed in Chapter Ten.) Ethical Principles in Question Asking Discussions of ethical problems in survey research have centered on three principles: the right of privacy, informed consent, and confi-
Slide 33: THE SOCIAL CONTEXT OF QUESTION ASKING 13 dentiality. Survey research is intrusive in the sense that the privacy of respondents is violated when they are selected to participate in the survey and then asked a series of questions. It is critically important to be aware of respondents’ right of privacy. Westin (1967, p. 373) defines right of privacy as “the right of the individual to define for himself, with only extraordinary exceptions in the interest of society, when and on what terms his acts should be revealed to the general public.” For the purpose of survey research, we would extend Westin’s definition to include attitudes, opinions, and beliefs, in addition to actions. Why the Right of Privacy Is Not Absolute Several aspects of the right of privacy have implications for the ethics of survey research. First, privacy is not viewed as an absolute right. The interests of society are recognized in extraordinary circumstances as sometimes justifying a violation of privacy, although the presumption is in favor of privacy. Second, the right of privacy with regard to information refers to people’s right to control data about themselves that they reveal to others. They can certainly be asked to reveal data about themselves that may be highly sensitive, but they have the right to control whether they voluntarily answer the question. There is no presumption of secrecy about a person’s activities and beliefs. Rather, people have the right to decide to whom and under what conditions they will make the information available. Thus, the right of privacy does not prevent someone from asking questions about someone else’s behavior, although under some conditions it may be considered rude to do so. The right of privacy does, however, protect respondents from having to disclose information if they do not wish to. And it requires that information revealed under conditions of confidentiality must be kept confidential. With regard to confidentiality of information, norms may vary from situation to situation. In some cases, such as with medical or legal information, explicit authorization is needed to communicate the information to a third party (for example, “You may tell X”). In
Slide 34: 14 ASKING QUESTIONS other situations, such as during ordinary conversations, the implicit norm is to permit communication about the contents of the conversation to third parties unless there is an explicit request for confidentiality (for example, “Keep this confidential”). One of the reasons for routine explicit assurance of confidentiality in research interviews is to overcome the natural similarity between research interviews and everyday conversations with strangers, which have the implicit norm of nonconfidentiality. What’s Informed Consent? The term informed consent implies that potential respondents should be given sufficient information about what they are actually being asked and how their responses will be used. The intent is for them to be able to judge whether unpleasant consequences will follow as a result of their disclosure. The assumption is that people asked to reveal something about themselves can respond intelligently only if they know the probable consequences of their doing so. The standards by which procedures for obtaining informed consent are evaluated usually refer to the risks of harm to respondents who provide the requested information or participate in a particular research activity. What it means to be “at risk” thus becomes crucial for a discussion of the proper procedures for obtaining informed consent. When is consent “informed”? Unfortunately, there does not appear to be agreement on the answer to this question. It is generally thought that the amount of information supplied to the respondent should be proportional to the amount of risk involved. You must ask yourself, then: “How much risk is actually involved in the research? How completely can I describe the research without contaminating the data I am trying to obtain? How much will a typical respondent understand about the research project? If respondents do not understand what I am telling them, is their consent to participate really informed?” These questions and variations on them plague researchers as they try to define their obligations to respondents.
Slide 35: THE SOCIAL CONTEXT OF QUESTION ASKING 15 The Important Role of Institutional Review Boards Research conducted today within a university or medical research setting that receives support from federal grants requires that protocols for informing research participants about their participation risks and for ascertaining their informed consent must be approved by an Institutional Review Board (IRB) composed of both peers and lay members of the community. Although the motivating force to establish IRBs was to ensure that participants in biomedical experiments or clinical trials were adequately informed about the risks to their health in taking part in the experiment, the review procedures have been extended little by little to include all research involving human participants whether it involves health or not and whether it is supported by the federal government or not. Many IRBs now require review even of pilot tests and focus groups that are intended to pretest a survey instrument prior to its use in the field. Fortunately, most IRBs have a special procedure to expedite review of protocols for surveys that do not involve sensitive topics or that involve respondents who are not in a special risk category. (Respondents who might be in a special risk category include minors or those participating in drug treatment programs.) In some cases, however, IRBs whose members are not familiar with social research have placed requirements on survey researchers for written consent forms that are more appropriate for biomedical research projects than for population-based surveys. As noted earlier, obtaining an interview requires a delicate negotiation between the interviewers (and researcher) and the selected respondents. The negotiation must balance privacy and confidentiality issues against the benefits of participating in the survey. If the requirements for elaborate signed consent forms become excessive or inappropriate to the risks of participating, participation rates will fall to levels that may not be high enough to justify the research. Respondents in the vast majority of surveys are not “at risk,” where risk is thought of as the possibility that harm may come
Slide 36: 16 ASKING QUESTIONS to respondents as a consequence of their answering questions. However, some surveys do ask about illegal or socially disapproved of behavior that could constitute a nonphysical risk. In such cases, respondents’ answers, if revealed to others, might result in social embarrassment or prosecution. For those surveys extra care is taken to ensure confidentiality and security of the responses. In other instances a survey may contain questions that will make some respondents anxious and uncomfortable. A recent study asked World War II veterans to respond to questions regarding how their combat experience influenced subsequent attitudes and longterm behaviors (Sudman and Wansink, 2002). Even though the events occurred more than fifty years ago, many individuals chose to skip the section related to their combat experiences. If these studies are being conducted with personal interviews, carefully and thoroughly training interviewers can help remove such anxiety and discomfort. Professional interviewers are excellent at creating an environment in which respondents can talk about personal matters without embarrassment. In fact, this professional, nonjudgmental questioning is one of the ways that survey interviews differ from ordinary conversations. If questions elicit anxiety from respondents for personal reasons, however, the interviewer can do little other than inform the respondent as fully as possible about the survey’s subject matter. Interviewers typically inform respondents of the general purpose and scope of the survey, answering freely any questions the respondents ask. If the survey contains questions that might be sensitive or personal, respondents should be told that such questions will be in the interview schedule and that they do not have to answer them if they do not wish to do so. Written consent is not typically obtained because it is usually clear that participation is voluntary. If the interviewer will have to obtain information from records as well as directly from the respondent—for example, if a respondent’s report about an illness must be checked against hospital records—written permission to consult the records must be ob-
Slide 37: THE SOCIAL CONTEXT OF QUESTION ASKING 17 tained. For many interviews with minors, written permission from parents or legal guardians must be obtained. Helping Guarantee Anonymity Does informed consent imply that the respondent must be explicitly told that participation in the survey is voluntary? Many practitioners feel that informing the respondent of the general nature of the survey and assuring confidentiality make it sufficiently clear that participation is voluntary. In some cases, informing respondents about the general nature of the survey can be as simple as saying, “This survey will ask you about your shopping behaviors” or “We will be asking you about your attitudes toward various leisure activities.” To go beyond the ordinary norms of such situations is to raise the suspicions of respondents that something is not quite right about this survey. For example, Singer (1978) found that even a request for a signature reduced the response rate for the questionnaire as a whole. In another study (Wansink, Cheney, and Chan, 2003), a split-half mailing that asked five hundred people to write their name and address on the back of a survey yielded a 23 percent decrease in response. Under certain circumstances merely asking a question might be harmful to respondents. For example, if you were conducting a follow-up survey of individuals who had been in a drug or alcohol rehabilitation program, the very fact that respondents were approached for an interview would indicate that they had been in the program. If they did not want that fact known to family or friends, any contact and attempt to ask questions might give rise to mental stress. Here problems of privacy, consent, and confidentiality are thoroughly entwined. In such cases it is important to protect the respondents’ privacy, to ensure that they will not be “at risk,” and to keep information confidential. To do so, great attention must be given to research procedures to ensure the respondent (or his or her relationship with friends, families, or employers) is not harmed.
Slide 38: 18 ASKING QUESTIONS This attention needs to begin prior to the first attempt to contact respondents and must continue through to the completion of the research. Except in special cases of some surveys involving substance abuse and other topics collected under a “shield law,” individual responses to surveys are not protected from subpoena by law enforcement officials or attorneys if the individuals are involved in a lawsuit. The fact that the researcher has promised confidentiality to the respondents will not protect the researcher from having to produce the individual records if required by legal action. As a matter of prudence, judges often deny requests from attorneys or legal officers for access to individual records, but they balance the requirements of justice in each case against the public good of protecting the confidentiality of research records. The only way researchers can be sure to keep individual data confidential—if it is not protected by a shield law—is to destroy the names and addresses of respondents and any links between the responses and names. Unless the names and addresses are required for follow-up interviews in a longitudinal study, it is best to destroy as soon as possible any data that could potentially identify the respondent. In some cases, this can also include data on variables that could be used to infer an individual’s identity, such as birth dates, treatment dates, and other detailed information. In cases where names and addresses are needed for longitudinal studies, two separate files should be established, one for the names and one for the location data, with a third file containing the code necessary to link the two files. The identifier files can be kept in a separate and secure site that has the maximum protection possible. In one case, there was reason to expect that the identifier files might be subpoenaed and misused in a way that would reveal the identities of all individuals in the file. In this case, the identifier files were kept in a country where they are not subject to U.S. subpoena. The intent of such seemingly exceptional measures is to protect the privacy of respondents by making it as difficult as possible to link individual identifier data with the
Slide 39: THE SOCIAL CONTEXT OF QUESTION ASKING 19 substantive data. Besides protecting the trust under which the data were collected, this also helps avoid inadvertent disclosure and makes the cost of obtaining the linked data very high for those who might be fishing for something useful in a legal case. How Much Do Respondents Need to Know? Most survey researchers limit themselves to rather general descriptions of the subject matter of the survey. Most respondents’ refusals occur before the interviewers have had time to explain fully the purposes of the survey. For the vast majority of sample surveys, the question is not really one of informed consent but, rather, one of “uninformed refusal.” Participation in surveys is more a function of the potential respondents’ general attitude toward surveys than of the content of a specific survey. Sharp and Frankel (1981) found that people who refuse to participate in surveys are more negative about surveys in general, more withdrawn and isolated from their environment, and more concerned about maintaining their privacy, regardless of the purpose of the survey. Today, refusals may also occur simply because of an increased amount of perceived or actual time pressure. In sum, it is your ethical responsibility as a researcher to inform the respondent as fully as is appropriate about the purposes of the survey, to explain the general content of the questions, and to answer any questions the respondent may have about the nature of either the scholarship or the sponsorship of the research and how the data will be used. In addition, you should inform respondents about the degree to which their answers will be held confidential. Although you must make every effort to ensure that that degree of confidentiality is maintained, you must not promise a higher degree of confidentiality than you can in fact achieve. Thus, for example, if the conditions of the survey do not allow you to maintain confidentiality against subpoenas, you should not so promise your respondents.
Slide 40: 20 ASKING QUESTIONS The Research Question Versus the Actual Question Being Asked In discussing questionnaire development, we must distinguish between the research question and the particular questions that you ask respondents in order to answer the research question. The research question defines the purposes of the study and is the touchstone against which decisions are made about the specific individual questions to be included in the questionnaire. The research question is most often general and may involve abstract concepts that would not be easily understood by the respondents being surveyed. For example, you may want to determine the attitudes of the American public on gun control, the effects of a particular television program on health information and health practices of those who view it, or whether an increase in automation is resulting in an increase in worker alienation. Articulating the Specific Purpose of the Study Regardless of whether the purpose of the research is to test a social scientific theory or to estimate the distribution of certain attitudes or behaviors in a population, the procedures for questionnaire construction are similar. First you will need to identify the concepts involved in the research question. Then you will formulate specific questions that, when combined and analyzed, will measure these key concepts. For example, if you are interested in the attitudes of potential voters toward a particular candidate, you will have to decide which attitudes are important for the topic at hand: attitudes about the particular positions the candidate holds, attitudes about the candidate’s personality, or attitudes about the candidate’s likability. The more clearly formulated and precise the research question, the more easily the actual questions can be written and the questionnaire designed. The process of trying to write specific questions for a survey helps clarify the research question. When there are ambiguities in
Slide 41: THE SOCIAL CONTEXT OF QUESTION ASKING 21 question wording or alternative ways of wording questions, decisions about formulating questions must be consistent with the original purposes of the survey. Often the purposes themselves may not be very clear and must be further refined before a final choice can be made. For instance, if you were conducting a survey with the purpose of deciding whether a potential candidate should run for a particular office, you might be interested in how much respondents know about the person, what political views they identify with that person, and what they are looking for in a good candidate. In contrast, if you were conducting a survey for a candidate who had already declared her intention to run for office, you might be more interested in what respondents think about the candidate’s stand on particular issues and whether they intend to vote for that candidate. Writing Questions that Relate to the Purpose of the Study Even when surveys are being conducted on the same topic, very different questions might be asked depending on the specific purpose of the study. For example, most surveys ask about the educational level of the respondent. If, for the purposes of your survey, a grouping of respondents into three or four levels of education will suffice, then a simple question like “What is the highest grade you completed in school?” with three or four response categories may well serve the purpose. If, however, the purposes of your survey require that the educational level of the population be precisely estimated, you would need considerably more detail about education—making distinctions, for example, between degrees granted and years of education started but not completed. Because the way in which questions are asked is intimately tied to the purposes of the survey, there is no “standard” way to ask about personal characteristics, such as education and income. (See the discussion in Chapter Nine.) As a general rule, when constructing a questionnaire, you must continuously ask “Why am I asking this question?” and must, in each instance, be able to explain how the question is closely related to the research question that underlies the survey. Our training as
Slide 42: 22 ASKING QUESTIONS researchers has always led us to believe that more information is good. Unfortunately, it becomes costly if we lose our focus when constructing a survey. The problem usually begins with someone saying, “Wouldn’t it be interesting to know. . . ?” The problem is that when the resulting cross-tabs, bar charts, or pie charts are presented, a great deal of time and money has been spent and we may not be much wiser than prior to the research. It is critical to keep focused on the basic research question. Suggestions for Beginners The process of writing questions is fun, and well-written questions can quickly engage the interest of the participants. Competition develops among the question writers to see who can come up with the cleverest or most interesting questions. Given our biases toward more information, a game of “Wouldn’t it be nice to know?” can quickly ensue, and soon there are many more questions than the budget can afford or than respondents can endure. Too often questionnaire writers are so caught up in the excitement of question writing that they jump rapidly into writing questions before they have adequately formulated the goals of the research and thoroughly understood the research questions. Many questionnaires constructed by inexperienced people look as if the researchers did not know what they were trying to find out until they saw what they had asked. To develop a good questionnaire, observe the following rules: 1. Resist the impulse to write specific questions until you have thought through your research questions. 2. Write down your research questions and have a hard copy available when you are working on the questionnaire. 3. Every time you write a question, ask yourself “Why do I want to know this?” Answer it in terms of the way it will help you to answer your research question. “It would be interesting to know” is not an acceptable answer.
Slide 43: THE SOCIAL CONTEXT OF QUESTION ASKING 23 Use Questions from Other Surveys It is always useful before creating new questions to search for questions on the same topic that have been asked by other researchers. This can justify your questions and provide an important point of comparison. In academic research, using validated scales is critical for research to be publishable in key journals. Yet satisfactory existing questions are unlikely to cover all the research questions of a study. Most questionnaires consist of some questions that have been used before and some new questions, although even the new questions may be adapted from earlier ones. Using existing questions will shortcut the testing process and may also allow you to compare results across studies. For studies done with similar populations and in similar contexts and where there is no reason to expect changes, using identical questions allows you to estimate response reliability. Over longer time periods or where changes are expected, using the same question permits estimates of trends. Some researchers have ethical concerns about using another person’s questions, but the replicating nature of social science research in general and survey research in particular not only permit but encourage the repetition of questions. Normally, no permission from the originator of the question is required or expected. You may, however, want to communicate with the question originator to learn whether there were any difficulties with the question that were not discussed in the published sources. If you want to use items from a questionnaire that has been copyrighted, permission from the publisher, and probably the payment of a small fee, would be required. Generally, in any given report, it will be important to acknowledge the source of any questions that are asked. However, researchers are becoming increasingly aware that simply replicating questions might not be so simple as it seems on the surface. Attention must also be paid to the context within which particular questions are asked, since responses to some questions are sensitive to
Slide 44: 24 ASKING QUESTIONS the context defined by the questions asked prior to them (Schuman and Presser, 1981; Sudman, Bradburn, and Schwarz, 1996). If you are interested in the trend in responses to a question over time, pay particular attention to the preceding questions asked in the studies where the question was previously used. (The order of questions in a questionnaire is discussed in Chapter Ten.) Once you start looking, you will be surprised at the variety of sources that can provide examples of earlier questions on a topic. The two major sources of survey questions are published material and data archives. Although we list a few of the major sources and archives, the list is intended to be suggestive rather than complete. Getting help from an available research librarian or information specialist can be very helpful. Finding Good Questions from Other Surveys We assume that a careful literature search has been conducted to help define the research questions. When a reference is a complete book, a copy of the questionnaire will often be included as an appendix. In journal articles, however, the questionnaire will usually be omitted due to lack of space. In this case it is appropriate to write to the author of the study and ask for a copy of the questionnaire. More general sources of questions include the following: Gallup, G. H. The Gallup Poll: Public Opinion, 1935–1971. (3 vols.). Gallup, G. H. The Gallup Poll. Public Opinion, 1972–1977. (2 vols.) Hastings, E. H., and Hastings, P. K. (eds.). Index to International Public Opinion, 1978–1979. National Opinion Research Center. General Social Surveys, 1972–2002: Cumulative Codebook. New York Times/CBS News polls, as indexed in The New York Times Index.
Slide 45: THE SOCIAL CONTEXT OF QUESTION ASKING 25 Opinion Roundup section of the Public Opinion Polls section of Public Opinion Quarterly. Robinson, J. P., Rusk, J. G., and Head, K. B. Measures of Political Attitudes. Robinson, J. P., and Shaver, P. R. Measures of Social Psychological Attitudes. (Rev. ed.) Roper Public Opinion Research Center. Survey Data for Trend Analysis: An Index to Repeated Questions in U.S. National Surveys Held by the Roper Public Opinion Research Center. Some of the largest American archives of survey research data are listed next. (Refer also to the Appendix for a list of the major not-for-profit survey research labs in North America and Europe.) There will normally be some charge for locating and reproducing questions and results. In addition, government, university, and other nonprofit survey organizations will usually make their questions and questionnaires available to others, even if they have no formal archives. Data and Program Library Service, University of Wisconsin, 4451 Social Science Building, Madison, WI 53706 Institute for Research in Social Science, Manning Hall, University of North Carolina, Chapel Hill, NC 27514 Inter-university Consortium for Political and Social Research, University of Michigan, Ann Arbor, MI 48106 (Institute for Social Research archives are at the same address.) National Opinion Research Center, University of Chicago, 6030 South Ellis Ave., Chicago, IL 60637 Roper Public Opinion Research Center, 341 Mansfield Road, Unit 1164, University of Connecticut, Storrs, CT 06269 Survey Research Center, University of California, Berkeley, CA 94720 Survey Research Lab, University of Illinois, Champaign, IL 61820
Slide 46: 26 ASKING QUESTIONS This search for existing questions sometimes becomes tedious and time-consuming, but it is time well spent. Even if you ultimately use only a few existing questions, the search generally helps you sharpen the research question and improve the quality of the new questions that you write. Consider the following caveats when adapting questions from other sources. Very small changes in wording or in the response categories offered can result in large differences in results. Within a year of each other, three polls (see Figure 1.1) asked representative samples of Americans about who they believed to be the greatest male athlete of the twentieth century (closed-ended), the greatest male or female athlete living at any point in the twentieth century (open-ended), and the greatest active athlete in the world of sports today (open-ended). Although all were taken within one year of each other, there is very little correspondence between the three. This underscores the importance of making certain any questions that are borrowed or replicated from another source specifically identify the issue that is of primary interest to your research question. Sources of Error in Responses Since questionnaires are designed to elicit information from respondents, the quality of a question can be measured by the degree to which it elicits the information that the researcher desires. This criterion is called validity. Directly measuring the validity of questions is often difficult and depends on the nature of the question. Different Types of Questions Have Different Errors We find it useful to divide questions into the following three groups: (1) those that ask about behavior or facts, (2) those that ask about knowledge, and (3) those that ask about psychological states or attitudes. Behavioral or factual questions ask about characteristics of people, things people have done, or things that have happened
Slide 47: THE SOCIAL CONTEXT OF QUESTION ASKING 27 Figure 1.1. Who is the World’s Greatest Athlete? NBC News/ Wall Street Journal, Sept. 9-12, 1999 (N = 1,010) Gallup/CNN/ USA Today, Dec. 20-21, 1999 (N = 1,031) “What man or woman living anytime this century do you think was the greatest athlete of the century, in terms of their athletic performance?” (open-ended) % The Gallup Poll, Aug. 24-27, 2000 (N = 1,019) “Which one of the following do you consider to be the greatest American male athlete of the 20th century?” (closed-ended) % “In your opinion, who is the greatest athlete active in the world of sports today?” (open-ended) % Michael Jordan Babe Ruth Muhammad Ali Jim Thorpe Jesse Owens Jackie Robinson Jack Nicklaus Johnny Unitas Mark McGwire Walter Payton Jackie Joyner-Kersee Tiger Woods Cal Ripken Other No Opinion, Not Sure, or None *1% or less apiece 35 13 11 11 10 7 2 1 n/a n/a n/a n/a n/a 1* 9 23 4 0 4 3 0 0 0 9 2 2 0 0 27* 26 4 0 0 0 0 0 0 0 3 0 0 30 2 26* 35
Slide 48: 28 ASKING QUESTIONS to people that are, in principle, verifiable by an external observer. That is, behavioral questions concern characteristics, events, or acts that are external to the individual and could be observed by a third party. (To say that they are in principle verifiable does not mean, of course, that it would be easy to verify them or, in some cases, that it is even legal or ethically permissible to verify them, such as with voting records or sexual behavior.) Questions about knowledge measure respondents’ knowledge about a topic of interest or their cognitive skills. In sample surveys, knowledge questions are often combined with attitude or behavior questions to gauge the saliency of an issue or the outcome of a program. Questions that have the form of knowledge questions are sometimes used as disguised attitude questions. More rigorous forms of measuring knowledge, as in knowledge tests, are frequently used to survey schooling outcomes. The field of psychometrics deals with the sophisticated statistical techniques for the reliable and valid measurement of knowledge. Discussion of these techniques is beyond the scope of this book. Researchers interested in the serious measurement of knowledge should consult with a psychometrician in developing their questionnaires. Questions about psychological states or attitudes are not verifiable even in principle, since states or attitudes exist only in the minds of the individuals and are directly accessible, if at all, only to the individuals concerned. Psychological states or attitudes are not available to an external observer. For behavior, the notion of validity has an intuitive meaning, as the value that would be agreed on by several external observers observing the same event. For attitudes, the intuitive meaning of validity is not clear. Should the measure of validity be what respondents tell about themselves in moments of privacy with their most intimate friends, or should it be what has a strong relationship to actual behavior? The answer lies more in one’s theoretical conceptualization of attitudes than in generally agreed-on criteria. Even though one may not have a clear idea about validity criteria for attitude questions, it is nonetheless certain that differing ways
Slide 49: THE SOCIAL CONTEXT OF QUESTION ASKING 29 of asking questions may produce quite different answers and that questions about some attitudes are more susceptible to questionwording differences than others. We do not yet know the detailed mechanisms that produce such changes, but we are beginning to understand the cognitive processes involved. (See Sudman, Bradburn, and Schwarz, 1996, and Tourangeau, Rips, and Rasinski, 2000, for a more complete discussion.) It is clear, however, that some attitudes are more variable in their measurement than others. The Difference Between Bias and Variability In our thinking about these issues, we have used the concept of response effect to include components of bias and variability. Bias refers to an estimate that is either more or less than the true value. Variability is measured by the susceptibility of measurements to differences in question wording. This variability is sometimes called the reliability of a measure, since random errors may arise from the form of the measurement itself (rather than from systematic error due to a sample bias or some other aspect of the measurement instrument). In order to clarify the sources of response effects, let us look at a particular behavioral question. A common question in surveys is “What was your total family income from all sources last year?” There is a true answer to this question, even though we may never know what it is since even income tax records, assuming that we had access to them, contain their own source of error. However, even though there is a true answer to this question, we may get an erroneous answer because the respondent simply forgot about certain amounts of income, particularly those from less obvious sources (such as dividends from a stock or interest on a savings account), or because the respondent may attribute income to the wrong year. The incorrect placement of events in a particular time period is called telescoping. In forward telescoping, the respondent includes events from a previous time period in the period being asked about; in backward telescoping, the respondent pushes events backward into a time period previous to the one being asked about. Forward
Slide 50: 30 ASKING QUESTIONS telescoping typically results in overreporting of events; backward telescoping typically results in underreporting. Both forward and backward telescoping may occur with the same frequency in a survey, so that the two may cancel each other out. However, studies show that forward telescoping is more common, resulting in a net overreporting of the telescoped material in most surveys. Motivated and Unmotivated Biases Another form of error would be the deliberate or motivated nonreporting of income that the respondent wishes to conceal—for example, illegal income or income not reported to the IRS. Another source of error arises from the deliberate overstating or understating of income in order to make an impression on the interviewer. Generally this type of error shows in income inflation, but some respondents, particularly in the upper income ranges, may deflate their reported incomes. Yet another source of error stems from the respondent’s failure to understand the question in the way the researcher intended. For example, the respondent may fail to report gift income, even though this type of income was intended by the researcher to be included. Finally, respondents may simply be ignorant of some income (perhaps income received by family members) about which they are asked to report. This rather involved collection of errors can be identified by four basic factors related to response error: memory, motivation, communication, and knowledge. Material may be forgotten, or the time at which something happened may be remembered incorrectly. Respondents may be motivated not to tell the truth because of fear of consequences or because they want to present themselves in a favorable light. Respondents may not understand what they are being asked, and answer the question in terms of their own understanding. Finally, they may just not know the answer to the question, and answer it without indicating their lack of knowledge. In the chapters that follow, these factors and the way they affect the business of asking questions will be explored in greater detail.
Slide 51: THE SOCIAL CONTEXT OF QUESTION ASKING 31 Additional Reading Consult the references listed in this chapter (in the section on “Suggestions for Beginners”) for additional examples of questionnaire wordings and their effect on responses. The Polls section of Public Opinion Quarterly is especially useful. It summarizes questions on different topics in each issue. In addition, the following readings may be useful. The Psychology of Survey Response (Tourangeau, Rips, and Rasinski, 2000) and Thinking About Answers (Sudman, Bradburn, and Schwarz, 1996) present conceptual frameworks and extensive scientific evidence for understanding response effects in surveys. They are recommended to the reader who wishes to pursue the conceptualization and literature behind the recommendations given in this book.
Slide 53: Part Two Tactics for Asking Questions
Slide 55: Chapter Two Asking Nonthreatening Questions About Behavior The most direct and probably the most common questions asked of respondents relate to their behavior. It is hard for a novice question writer to see any problems with a question like “Do you own or rent your place of residence?” or “What brand of coffee did you buy the last time you purchased coffee?” Nevertheless, such questions are not so simple and straightforward as they might first appear. Questions about behavior may be viewed as threatening and may result in biased reports. Clearly, it is more difficult to ask a question about child abuse or spousal abuse than about owning a television set. But even questions about such topics as voting in a recent election or owning a library card may be threatening enough to disrupt the smooth interaction between the interviewer and the respondent. This interruption may come about because the question causes respondents some discomfort or because the respondents believe that the truthful answer to the question will put them in a bad light and cause the interviewer to think less well of them. We defer the topic of asking threatening questions to the next chapter and limit the discussion here to questions that are not threatening (or, at least, not very threatening). Such questions may relate, for instance, to work activities, ownership or purchases of consumer goods, some forms of health-related behavior, social interactions with others, or vacation and travel behavior. Questions on household composition, income, employment, and other demographic characteristics might be discussed here but are deferred to Chapter Nine, where standard wordings are suggested. 35
Slide 56: 36 ASKING QUESTIONS As we shall see later, both threatening behavior questions (Chapter Three) and attitude questions (Chapter Five) are very sensitive to question wording. Although nonthreatening behavior questions are less sensitive to wording changes than other questions, they are influenced by comprehension and memory. When these questions are correctly comprehended, the most serious problem with nonthreatening behavioral questions is that human memory is fallible and depends on the length and recency of the time period and on the saliency of the topic. In this chapter we discuss what is known about memory errors and then suggest a series of strategies for reducing these errors. Checklist of Major Points 1. Decide whether the question is or is not threatening. If threatening, see also Chapter Three. 2. When asking a closed-ended question about behavior, make sure that all reasonable alternative answers are included. Omitted alternatives and answers that are lumped into an “Other” category will be underreported. 3. Aided-recall procedures may be helpful if the major problem is underreporting of behavior. 4. Make the question as specific as possible. More reliable information is obtained when you ask about behavior in an exact time period instead of asking generally about respondents’ usual behavior. If the goal, however, is simply to group respondents into categories rather than precisely measure their behavior, such questions do not have to be so precisely worded. 5. The time period of the question should be related to the saliency of the topic. Periods of a year (or sometimes even longer) can be used for highly salient topics, such as purchase of a new house, birth of a child, or a serious auto accident.
Slide 57: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 37 Periods of a month or less should be used for items with low saliency, such as purchases of clothing and minor household appliances. Periods that are too short, however, should be avoided, since forward telescoping (remembering the event as having occurred more recently than it did) can cause substantial overreporting of behavior. 6. For regular, frequent behavior, respondents will estimate the number of events by using the basic rate they have stored in memory. Accuracy of these estimates can be improved by asking about exceptions to respondents’ regular behavior. 7. The use of secondary records (where available), household observation, and bounded recall will reduce or eliminate telescoping and also improve the reporting of detailed information. 8. Where detailed information on frequent, low-salience behavior is required, providing diaries will result in more accurate results than memory. 9. Use words that virtually all respondents will understand. Do not use special terms or vocabulary unless all members of the sample would be expected to know them or the term is explained in the question. 10. Increasing the length of the question by adding memory cues may improve the quality of reporting. Do not assume that the shorter questions are necessarily better. 11. Recognize that, for nonthreatening behavior, respondents will generally give more accurate information about themselves than about relatives, friends, or coworkers. If cost is a factor, however, informants can provide reasonably accurate information about others, such as parents about children, and spouses about each other. Ten Examples of Behavioral Questions We start with examples of questions used by various government and other survey agencies for collecting information about behavior.
Slide 58: 38 ASKING QUESTIONS These questions represent the work of professional questionnaire designers. All have undergone careful review and pretesting. Nevertheless, they are not immune from the memory and other problems that we discuss later in this chapter. Outdoor Recreational Activities Figure 2.1 illustrates a series of questions about outdoor recreational activities. Part 1, which asks only whether the respondent ever did an activity during the last twelve months, is considerably easier to answer than Part 2, which asks for the number of times the respondent participated. Limiting participation to that in the State of Illinois makes the question still more complex. As we shall discuss later in the chapter, it is highly likely that respondents who frequently engage in an activity will not count individual episodes but will estimate. The period for activities is extended to a year because many of these activities are seasonal; a survey conducted in the winter would get no data on summer sports. Jogging There are several interesting wording uses in the Gallup question on jogging, shown in Figure 2.2. The use of the words “happen to” in the question “Do you happen to jog, or not?” is intended to reduce or eliminate social desirability biases. Although jogging appears to be a nonthreatening topic, some respondents who do not jog might be tempted to report that they did, because jogging is popular and associated with health and fitness. Similarly, adding the words “or not” is intended to give equal weight to both the positive and the negative answer. Although the responses to this question from the 1996 Gallup Poll might not differ substantially from those to the simpler question “Do you jog?” the additional words are intended to ensure the accuracy of the results.
Slide 59: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 39 Figure 2.1. Outdoor Recreation Survey. 1. First, I’d like to get a general idea about the specific kinds of things you do for recreation or to relax. I have a list of activities people sometimes do. Please think back over the past month, since . As I read each activity, please tell me whether or not you have done it this past month. Did you . . . Yes A. Go to a movie? B. Dine at a restaurant for pleasure? C. Go window shopping? D. Go to a theater or concert? E. Go on a picnic? F. Go hunting or fishing? G. Read for pleasure? H. Take a ride in an automobile for pleasure? I. Do gardening for pleasure? J. Participate in a civic or religious organization or club? K. Go for a walk or a hike? L. Go to a professional, college, or high school sports event? No 2. Now, I have some questions about sports. Please think back over the past year, since . Did you . . . (Enter date 1 year ago today) Yes A. Play badminton? B. Play basketball? C. Go bowling? D. Play football? E. Play golf? F. Play racketball, handball, paddleball, or squash? G. Play softball or baseball? H. Swim? I. Play tennis? Source: National Opinion Research Center, 1975. No
Slide 60: 40 ASKING QUESTIONS Note also the explanations given in the body of Question 1. Respondents may not know what is meant by the word “regularly.” Some might assume that it meant monthly or weekly, and some might ask the interviewer to clarify the word, which could then force the interviewer to decide what the word meant. By specifying “on a daily basis,” the question removes or reduces the uncertainty. Respondents who miss an occasional day may still be uncertain, but most respondents will not be. Also, in earlier surveys some respondents had answered “yes” to this question because they believed that their job helped to keep them physically fit. By excluding work “at a job,” the question makes it clear that only non-work-related activities are to be considered here. Health Services Figure 2.3 presents a condensed series of questions on the source and on the frequency of medical care (Survey Research Laboratory [SRL], 1993, 1978). (Attitudinal questions that were part of this series have been omitted.) The first question asking about visits to a medical doctor in the last year is widely used and seems straightforward, but it may be difficult for some respondents to know what is meant. Should they or shouldn’t they include visits to the doctor’s office for an allergy shot? Does it matter if the shot is given by the doctor or a nurse? The series of questions about the usual source of medical care does not directly ask about one or more specific events; instead, the respondent is asked to first perform a series of memory tasks and to then perform a series of comparison and averaging tasks. Thus, these questions appear to be difficult. Nevertheless, virtually all respondents were able to answer these questions, and the answers were sufficiently accurate to distinguish between respondents who had medical care readily available, those who had difficulty in obtaining care, and those who had no source of care.
Slide 61: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 41 Figure 2.2. Questions on Exercise. 1. Aside from any work you do here at home or at a job, do you do anything regularly—that is, on a daily basis—that helps you keep physically fit? Yes No 2. a. Do you happen to jog, or not? Yes No b. On the average, how far do you usually jog in terms of miles or fractions of miles? ________ miles Source: Gallup, 1978. Figure 2.3. Questions on Health Care. 1. During the last year, how many times did you see or talk to a medical doctor? times 2. Is there one particular person or place where you usually go for health care? Yes No (Skip to Q. 7.) 3. Have you been using this person or place as your usual source of health care for . . . Less than 6 months 6 months to 1 year More than 1 year but less than 3 years 3 to 5 years, or More than 5 years? Source: Survey Research Laboratory, 1993.
Slide 62: 42 ASKING QUESTIONS Household Health Diary Another procedure for obtaining health information is the use of a diary for recording events as they occur. Figure 2.4 illustrates a sample page from such a diary, including instructions, and sample entries inserted in the blanks. The diary also includes sections on “felt ill but went to work or school,” “visited or called a doctor,” “went to a hospital,” “obtained medical supplies,” and “paid doctor or hospital bills.” Although it would have been possible to ask about the details of the illness, such as why did the person feel ill, and what medicine or treatment was used, this information would be difficult to recall, especially for minor illnesses such as colds and headaches. Childrearing Practices Two comments can be made about the questions on childrearing shown in Figure 2.5. The first question is an open-ended, fieldcoded question (SRL, 1978). That is, respondents are not given the answers, but the interviewers have a list of categories into which to put the answers. (If the response is ambiguous, this procedure may introduce an additional source of error. This problem is especially important for attitude questions. Field coding is discussed in greater detail in Chapter Six.) Note that multiple answers are allowed but are not actively sought. Question 4 is a two-part question with skip instructions. The B part would be asked only if a “yes” is obtained in part A. Both the numbering and the skip instructions help guide the interviewer in asking questions of the respondents. Religious Practices Figure 2.6 illustrates that the Gallup Poll’s wordings on religious questions (2001) are similar to its wordings on the jogging question in Figure 2.2. It might be argued that membership in a church or
Slide 63: Figure 2.4. Household Health Diary. STAYED HOME FROM WORK OR SCHOOL OR COULD NOT DO USUAL HOUSEHOLD TASKS List all illnesses during this month to all household members who had to stay home from school, work, or could not do their usual job. If the same person starts off a little sick, but goes to work for two days and then stays home for two more days until he is recovered, you would report the first two days on page 5 and the last two days on page 3. SAMPLE What medicine or treatment was used? (Check one) Did they stay in bed all or part of the day? (Check one) Prescription None (Name, if known) Date first stayed home Why did they stay home? (Headache, cold, cramps, sprained ankle, etc.) Yes No Date resumed usual activities Who in the family? (First name) If other, what? Oct. 7 Flu Stomach cramps X X Dislocated shoulder X Oct. 9 John Aspirin X Plaster cast Oct. 13 Oct. 14 Mary Oct. 14 Oct. 19 John Jr. Source: Survey Research Laboratory, 1976.
Slide 64: 44 ASKING QUESTIONS Figure 2.5. Questions on Childrearing. 1. Where does your son/daughter regularly play or spend his/her free time? (Check all codes that apply.) At home In school In someone else’s house Just outside the house or in the yard In the street In a playground or park In a community building or community center Other (Specify) Don’t know 2. Does your son/daughter have a place at home where he/she can read or study in quiet? Yes No 3. Do you have any special time you set aside for being with children? Yes No 4. a. Do any of the following ever take care of your children? Neighbors Relatives Friends Teenagers Daycare center Nursery school Something else (Specify) (If all “No,” skip to Q. 7.) Yes Yes Yes Yes Yes Yes No No No No No No b. In an average week, how many hours are your children/is your child taken care of by someone other than yourself/you or your husband? hours Source: Survey Research Laboratory, 1978.
Slide 65: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 45 synagogue is a deliberate event and does not just happen. The same question asked in the General Social Survey simply asks, ”Are you, yourself a member of a church or synagogue?” Readers may wonder whether religion is a sensitive topic. For several decades the U.S. Census and other government sample surveys have not asked about religion because of concerns about the separation between church and state. Nevertheless, nongovernmental survey organizations have uniformly found that religion is not a sensitive topic and that reports of religious behavior are easy to obtain. In behavior questions the word you may often be confusing, since it may refer to the respondent or to all the members of the household. To avoid this confusion, use of “you, yourself” is often helpful when there may be ambiguity. Figure 2.6. Questions on Religion. A. Do you happen to be a member of a church or synagogue, or not? Member Not a member (Skip to Q. 2) B. Did you, yourself, happen to attend church or synagogue in the last seven days? Yes No Source: Gallup, 2001. Lawyers’ Survey Special problems arise in non-household surveys. The lawyers’ survey (Figure 2.7) was conducted by mail. This may well be an advantage for questions such as 3B and 3C, which ask for information on number of attorneys and other employees in the firm. In large firms the respondent would probably not have this information at hand and would need to spend a little time getting the count. Many business surveys are done by mail, so that respondents have a chance to collect the information. An alternative is to
Slide 66: 46 ASKING QUESTIONS Figure 2.7. Questions in Lawyers’ Survey. 1. In what year were you first admitted to the practice of law in any state? 2. a. Are you currently engaged in the practice of law? Yes, in private practice (Go to Q. 3a.) Yes, in nonprivate practice (Answer Q. 2b.) No, retired (Go to Q. 4.) No, in non-lawyer occupation (Go to Q. 4.) b. Which one of the following best describes your legal occupation? Business legal staff Government attorney Legal aid attorney or public defender Member of the judiciary Law faculty Other (Specify) (If not in private practice, go to Q. 4.) 3. a. Are you a sole practitioner, a partner, a shareholder, or an associate? Sole practitioner Partner or shareholder Associate b. How many other attorneys practice with your firm? (1) (2) Partners or shareholders Associates c. How many employees other than attorneys work for your firm as . . . (1) (2) (3) Secretaries? Legal assistants/Paralegals? Other? Source: Survey Research Laboratory, 1975.
Slide 67: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 47 send the questionnaire ahead by mail, so that necessary information may be collected, but to obtain the final answers in a personal interview so that ambiguous answers can be clarified. This survey uses specialized language such as “sole practitioner,” “partner,” “associate,” and “paralegals.” The specialized language causes the lawyer respondents no difficulty, although these are not meaningful terms to most non-lawyers. Farm Innovation Study The same use of specialized language is seen in Figure 2.8, dealing with farm practices (SRL, 1974). Again, these terms did not cause the surveyed farmers any serious difficulties. The most problematic questions in this series are those asking “How many years ago did you first do (have) this?” Farmers who have been following these practices for many years will have trouble remembering the beginning date unless it corresponds to an important anchor point, such as the year the respondent started farming this particular land. It should be possible, however, to distinguish between farmers who adopted a practice in the last year or two and those who adopted it more than ten years ago. Business Expenditures Sometimes the questions ask for more specificity than respondents can provide. Figure 2.9 gives such an example from the 1997 Economic Census. In the survey used to generate this form, the representative of a hotel was asked to report sales for detailed merchandise and receipt lines, such as for distilled spirits, wine, and beer and ale. Many hotels do not keep records at this level of detail and are unable to report this information, even though estimation is permitted. Both the questionnaire writer and the data analyst (if these are not the same person) must take a balanced view to questions that put such a substantial strain on the respondent’s memory or records, even when the results are aggregated. On the one hand, questions
Slide 68: 48 ASKING QUESTIONS Figure 2.8. Questions on Farm Practices. 1. Did you operate a farm last year? Yes No (End interview) 2. Farmers often find that some farm practices are more suitable for their own farm than other practices. Here are some practices we’d like to ask you about. (If Yes) How many years ago did you first do (have) this? Yes a. Do you use the futures market for selling grain? b. Do you dry corn on the farm? c. Do you use forward contract to sell crops? d. Do you have narrow crop rows, 36" or less? e. Do you use a large planter, 6 or 8 rows? f. Do you have a chisel plow? g. Do you use extension or USDA economic outlook information in planning farm business? h. Do you have a program to regularly test the soil to determine fertilizer applications? i. Do you keep farm records for reasons other than income tax? j. Do you use reduced tillage? 3. a. Do you use contour farming? Yes No (Skip to Q. 4.) b. How many years ago did you first do this? No c. Have you ever received money from the government for using contour farming? Yes No Source: Survey Research Laboratory, 1974.
Slide 69: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 49 Figure 2.9. 1997 Economic Census: Traveler Accommodations.
Slide 70: 50 ASKING QUESTIONS Figure 2.9. 1997 Economic Census: Traveler Accommodations, continued.
Slide 71: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 51 Source: U.S. Bureau of the Census, 1997.
Slide 72: 52 ASKING QUESTIONS should not be summarily omitted because precise information cannot be obtained; ballpark information can sometimes be very valuable. On the other hand, analysts should avoid making precise analyses of such loose questions. It is a serious but unfortunately common error to use powerful multivariate procedures carried to three decimal points with questions where even the first significant figure is in doubt. Consumer Expenditure Survey Shown as Figure 2.10 are questions on ownership and purchasing of major household equipment items. (Only the first page is shown.) This was a panel study, and these questions were asked twice, one year apart. Asking questions at two different time periods has the major advantage of reducing error in the date of purchase, by a procedure called bounded recall. (Bounded recall will be discussed in greater detail later.) Note also that the accuracy of reports about ownership is increased because the interviewer and the respondent have the opportunity to examine both furniture and appliances. On this survey and on similar surveys, researchers are not interested merely in ownership or possession but in information more difficult to recall, such as that involving brand and price. How to Tell if a Question Is Threatening There is no standard method to determine whether a question is threatening or not. Some questions that are not threatening in general may be threatening to particular individuals for idiosyncratic reasons; they might remind the respondent of a recent painful event or they might be mistakenly interpreted as referring to something that is unique to that individual. The best we can do is determine whether a question is likely to be threatening to a large number of respondents. The easiest way to determine the threat of a question is to ask ourselves whether we believe respondents will feel there is a right or wrong answer to it. Certain behav-
Slide 73: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 53 iors are seen by many people as socially desirable and therefore may be overreported. Examples follow. • Being a good citizen Registering to vote and voting Interacting with government officials Taking a role in community activities Knowing the issues • Being a well-informed and cultured person Reading newspapers, magazines, and books and using libraries Going to cultural events such as concerts, plays, and museum exhibits Participating in educational activities • Fulfilling moral and social responsibilities Giving to charity and helping friends in need Actively participating in family affairs and childrearing Being employed In contrast, the following are some examples of conditions or behavior that many people underreport in an interview: • Illnesses and disabilities Cancer Sexually transmitted diseases Mental illness • Illegal or contra-normative private behavior Committing a crime, including traffic violations Tax evasion Drug use • Consumption of alcoholic products • Sexual practices
Slide 74: Figure 2.10. Questions on Major Household Items.
Slide 75: Source: U.S. Bureau of the Census, 2001.
Slide 76: 56 ASKING QUESTIONS Many behavioral questions, however, are not at all threatening, or are only mildly threatening. Of the questions given in the previous examples, only a few (those dealing with childrearing in Figure 2.5) might be considered threatening, and even here the threat may not be serious. In some ways, social changes over the past several decades have made the survey researcher’s task easier. It is now possible to ask questions about cancer, drug use, and sexual behavior that could not have been asked earlier. Only a few respondents will refuse to answer these questions. Unfortunately, this does not mean that such questions are no longer threatening. Not all respondents will find a particular question threatening. Thus, a question about smoking marijuana will not be threatening to those who have never smoked or to those who feel that there is absolutely nothing wrong with smoking marijuana. It will be threatening, however, to respondents who smoke but are afraid that the interviewer will disapprove of them if they admit it. If you are in doubt about whether a question is potentially threatening, the best approach is to use previous experience with the same or similar questions. If no previous experience is available, a small pilot test can be informative. (See the discussion in Chapter Eleven. If the question is threatening or possibly threatening, see Chapter Three.) Eight Ways to Make Behavioral Questions Easier to Answer In the past decade we have gained a better understanding of the methods respondents use to answer questions about behavioral frequencies and numerical quantities, such as “How many times have you done (behavior) in the past two weeks?” or “How many aunts, uncles, and cousins do you have?” It is now well recognized that for many such questions respondents do not attempt to answer by counting individual episodes or units. Instead they often simply
Slide 77: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 57 make an estimate based on rates that are either stored in memory as schema or computed on the spot from a sample of available data. A general finding is that as the number of experiences of an event increases above five, respondents are more likely to estimate than to count (Blair and Burton, 1987). When behaviors are regular and similar, such as brushing one’s teeth or eating breakfast, estimation will result in more accurate responses than counting (Menon, 1997). The selection of the time period influences whether respondents count or estimate. Data users unfamiliar with cognitive processes often believe they can obtain much more information by increasing the length of the time period that a question covers, but this belief is illusory. If the behavior is frequent, irregular, and relatively unimportant, such as making a telephone call or buying gasoline for one’s car, respondents asked about a short time period will simply count and report the number of events retrieved. Respondents asked about a longer time period, will typically count for a short time period and then compute an answer based on this rate. Not only does the longer time period not provide additional information, it may increase the possibility of a computation error when the respondent is required to extrapolate. If the behavior is regular, respondents will already have a rate stored in memory and will simply retrieve this rate and apply it to whatever time period is specified. It is obvious that increasing the time period for regular behaviors has no effect on the amount of data obtained. For example, if respondents are asked how many times they brush their teeth in a given period of time, they would simply multiple their daily rate by the number of days in the time period they are asked to report. Only for infrequent, irregular behavior, such as buying consumer durables or going to the doctor, does increasing the length of the time period increase the amount of information retrieved. There are eight proven methods for improving the quality of reporting if respondents count.
Slide 78: 58 ASKING QUESTIONS Use Aided Recall In its most general sense, an aided-recall procedure is one that provides one or more memory cues to the respondent as part of the question. The questions in Figure 2.1 illustrate one form of aided recall. Rather than asking “What do you do for outdoor recreation?” the questions focus on specific activities and sports. Another form of this method is to put examples into the question, such as “How many organizations do you belong to—for example, unions, churches, fraternal organizations?” Similarly, respondents may be shown a card containing a list of books, magazines, and newspapers and asked which they have read in the past month. Aided recall may also be used with knowledge questions and with cards listing well-known persons, products, or organizations. This use is discussed in Chapter Six. A final form of aided recall is the household inventory conducted jointly by the respondent and the interviewer. These household inventories can be used to determine the presence of furniture, appliances, books and magazines, and goods such as food, soap, and cleaning products. Unless the product has been totally consumed, its presence is a memory aid. Aided-recall procedures produce higher levels of reported behavior than unaided procedures do (Sudman and Bradburn, 1974), since they can help respondents remember events that would otherwise be forgotten. Precautions When Using Aided Recall. Certain precautions must be observed, however, when aided recall is used. First, the list or examples provided must be as exhaustive as possible. As shown in general research on memory and in magazine readership and television viewing studies, behaviors not mentioned in the question or mentioned only as “Other (Specify)” will be substantially underreported relative to items that are mentioned specifically. If your questions concern media, products, and organizations, lists are almost certainly available from published directories. For
Slide 79: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 59 other types of behaviors, where outside lists are not available, earlier studies may provide information on the types of behaviors to include on the list. If such studies are not available, you would have to conduct a pilot study to obtain the necessary information. It is usually a mistake for a single researcher or even a group of researchers to develop a list of behaviors based only on personal experience. Personal experience is limited, and the inevitable consequence of relying on it is an incomplete and flawed listing. If the number of alternatives in a category is too great, your list may be restricted to a limited number of the most likely alternatives. Unfortunately, no estimate can then be made of the excluded behaviors. You could also include an “All Other” category in such aided-recall questions. Such a category is useful for rapport building because it gives respondents who otherwise would not have been able to respond positively an opportunity to answer. However, the data from this “All Other” category cannot be combined with the listed data. Moreover, if the list is not exhaustive, you cannot make an estimate of total behavior—although, by summing up only the listed behavior, you can make a minimum estimate. In some cases you can proceed in two stages, asking first about groups and then about specific cases. A list of all published magazines, for example, might be almost infinite in length. But you can group these into a dozen or so categories, giving examples for each category. For example, you might ask, “Do you regularly read any news magazines like Time or Newsweek? Any sports publications? Household or family magazines? Personal health and self-improvement magazines? Electronics or auto or hobby magazines?” This may be good enough if you merely want to code specific magazines into such groups anyway. But you can also ask for the names of particular magazines read within any or all categories the respondent reports reading. When a list becomes large, the order of the list may become important, especially when the respondent reads the list. Items at the top or at the bottom of a long list will be read or listened to
Slide 80: 60 ASKING QUESTIONS more carefully and will receive more positive responses than items in the middle. For long lists, careful researchers use two or more different forms and randomize the order of the items on both forms. Another procedure, shown in Figure 1.1, requires the interviewer to read all items to the respondent and obtain a “yes” or “no” answer for each item. This procedure is now widely used in telephone interviewing, where the respondent cannot be handed a card to read. It also has the advantage of removing or reducing list order effects, although both the interviewer and the respondent may become bored if the list is too long. Order effects are sensitive to the mode of administration. Because of primacy effects, items appearing early in the list are often over-selected when the questionnaire is administered in person with show cards or when it is self-administered. On the other hand, because of recency effects, items appearing at the end of the list are over-selected, particularly when the questionnaire is administered by telephone and the respondents can only hear the list read. Dealing with Long Lists. Another problem with aided recall develops from the use of long lists. Imagine respondents have been given a list of fifty activities and asked which of these they have done in a specified time period. If they have done none of these activities, the question is likely to make them uncomfortable, even if the topic is nonthreatening. They will feel that the interviewer expects at least some “yes” answers from among a long list of activities. Such respondents are likely to report some activities, either by deliberately fibbing or by unconsciously misremembering the date when a behavior occurred. You should anticipate this problem and avoid it by using two techniques. The first, illustrated in Figure 1.1, is to make the list so extensive that virtually all respondents will be able to answer “yes” to some items. The second way is to start with a screening question such as “Did you happen to have read any magazines in the past two weeks, or not?”—before showing the respondent a list of magazines.
Slide 81: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 61 The long list example typifies the most serious problem with aided recall—the implicit expectation that a respondent needs to provide positive responses. If a behavior is reasonably salient and the reporting period reasonably short, aided-recall procedures may lead to substantial overreporting and should not be used, or should be used only in conjunction with other procedures that reduce overreporting. (The exceptions to this rule are the socially undesirable behaviors discussed in Chapter Three, where aided-recall methods help compensate for the general tendency of respondents to underreport.) The short screener question—“Did you happen to read any magazines in the past two weeks, or not?”—may have the opposite effect. If such a screener is used several times in the interview, respondents may learn that they can skip out of a whole series of questions by saying “no.” In general, it is better to vary question formats where possible, to make the interview more engaging for the respondent and also to decrease the chances of respondent anticipation. Make the Question Specific One simple reason for making each question as specific as possible is to make the task easier for the respondent, which, in turn, will result in more accurate reports of behavior. General questions, if they are answered conscientiously, require substantial effort by the respondent. Consider a seemingly straightforward question such as “What brand of soft drink do you usually buy?” If the question is taken seriously, the respondent must first decide on the appropriate time period, and then what conditions to include. For instance, are purchases at work, in restaurants, at sporting events, and at movies to be included? Or are only store purchases for home use to be counted? The respondent must next decide on the meaning of the word you. Does it refer only to the respondent or to all members of the household? How are purchases by one household member for
Slide 82: 62 ASKING QUESTIONS other household members to be treated? A final question to be resolved is the definition of a soft drink. Are lemonade, iced tea, fruit punch, and mineral water to be included or not? A few respondents who are highly consistent in their behavior may nearly always choose the same brand. They can answer this question with little or no difficulty. But most respondents who buy several brands will have to do some cognitive work in order to answer this question. Some will respond with the first brand name that comes to mind. That is, they will change a behavior question into one dealing with brand awareness and salience. This leads to a substantial overreporting of purchases of widely advertised brands, such as Coca-Cola and Pepsi Cola. Only a few respondents will answer that they don’t know or ask the interviewer for more information. Thus, a small percentage of “don’t know” answers does not ensure that the question is answered accurately. As Payne (1951) points out, the researcher should behave like a newspaper reporter and ask the five W’s: who, what, where, when, and sometimes why. Whose Behavior? For behavior questions it should always be clear whether respondents are reporting only for themselves, for other household members, or for the entire household in total. The word you can be either singular or plural and is often a source of confusion. We suggest using “you, yourself” when information is wanted only from the respondent; “you or any member of this household” when the survey is attempting to determine whether any household member performed a given behavior; and “you and all other members of this household” when the survey is attempting to obtain total household behavior. Exactly the same system can be used if the interview takes place in an organizational or industrial setting. Just replace the word “household” with “company,” “firm,” or “organization,” as appropriate. What Behavior? Question 1 in Figure 2.2 illustrates a clarification of what behavior to report because it excluded all job-related activities. In a question about gasoline purchasing, you would want to
Slide 83: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 63 specify whether or not purchases while on vacation or other trips should be included. Similarly, in questions about food and drink consumption, it is necessary to specify whether out-of-home consumption is to be included or excluded. When Did it Happen? The “when” question should specify the time period by using actual dates instead of terms such as “last week” or “last month.” If an interview is conducted on June 28 and the respondents are asked about last month, some will consider the time period from June 1 to June 28 as the last month, and others will consider the period from May 28. Typical wordings that can be used are “In the past two weeks, that is, since June 14 . . .” or “in the past month (or thirty days) since May 21 . . .” It is generally less precise to ask “When was the last time you did something?” Even if respondents could remember accurately, this form gives equal weight to those who do something often and those who do it rarely. Analyses and conclusions based on such data are likely to be confusing and misleading. In addition, the memory task is more difficult for those who do it rarely, so that their answers are subject to much greater memory errors. Limiting the time period means that some (possibly many) respondents will report none of the specified behavior during the time period. This will bother researchers who are attempting to maximize the amount of information they get. However, from a perspective of total survey quality, it is better to minimize the number of erroneous or potentially erroneous responses. Asking Why and When Questions. This chapter is not the place to discuss “why” questions. It is also difficult to discuss “what” questions in general terms, since the “what” questions depend on the purpose of your research. You must have a clear idea of why your study is being done before you start to write questions. Although a few researchers are able to keep the aims of their study in mind without formal procedures, most—especially beginning researchers— cannot. Before you write any questions it is a good idea to put down
Slide 84: 64 ASKING QUESTIONS on paper the aims of the study, hypotheses, table formats, and proposed analyses. These aims should not become absolute, but they should provide some useful guidelines and boundaries. Even if you are clear on what is wanted, the respondent may still be uncertain, since respondents do not have your perspective on a topic. Belson (1981) demonstrates widespread misunderstanding of survey questions and such words as usually, have, weekday, children, young people, generally, regularly, and proportion. He hypothesizes that respondents will interpret broad terms or concepts less broadly than the researcher intended. He also suggests that respondents distort questions to fit their own situations or experience. Although one cannot ensure that all respondents will understand all questions exactly as intended, the use of specific questions will help reduce respondent differences in interpretation. If general or global questions are used, they should be tested to determine what respondents think they mean. Select an Appropriate Time Period to Ask About The basic idea to consider in determining a time period is that a person’s accurate recall of a behavior is directly related to the amount of time elapsed and to the salience of the behavior (Sudman and Bradburn, 1974). The more important the event, the easier it is for the respondent to remember. Although research on saliency is limited, there appear to be three dimensions that distinguish between events that are more and less salient: (1) the unusualness of the event, (2) the economic and social costs or benefits of the event, and (3) the continuing consequences of the event. Longer Time Periods for Highly Salient Events. Events that occur rarely in one’s life—such as graduating from high school, getting married, buying a house, having a baby, or having a serious motorcycle accident or surgery—are likely to be remembered almost indefinitely. Historical events can have the same saliency. Almost anyone who was old enough can remember exactly what they were
Slide 85: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 65 doing when Pearl Harbor was attacked, when President Kennedy was assassinated, or when the World Trade Center collapsed on September 11, 2001. In contrast, habitual events, such as all the things that one did at home and work, would be difficult to remember for even a day or two. In general, the greater the cost or benefit of an activity, the more one is likely to remember it. Winners of $100,000 in a state lottery will remember the details better than will the winners of $25. The purchase of a $500 microwave oven is easier to remember than the purchase of a $.69 potato peeler. Juvenile shoplifters will remember the time they were caught and forget the details of successful shoplifting efforts. Finally, some events result in continuing reminders that the event happened. The presence of a house, car, or major appliance is a reminder that the purchase was made. The presence of children is a reminder of their births. Many behavioral events are salient along two or three dimensions. Thus, buying a house is a unique event; it requires payment of a very large sum of money, and the presence of the building acts as a continuing reminder. On the other hand, the purchase of a food item is a low-cost, habitual act with no continuing consequences. Within this framework, memory about highly salient events is satisfactory for periods of a year or possibly more. Unfortunately, little work has been done on periods much longer than a year. However, for highly salient events, such as major accidents or illnesses, periods of two or three years appear to be possible. Periods of two weeks to a month seem to be appropriate for low-salience events. For behaviors of intermediate saliency, periods of one to three months are most widely used. Choosing an optimum time period does not mean that the data will be error free, but only that errors will be minimized if recall procedures are used. Longer Time Periods for Summary Information. When summary information is available, longer time periods can be used. Many respondents can give fairly reliable estimates of total medical expenditures, expenses for vacations, or income received in the past
Slide 86: 66 ASKING QUESTIONS calendar year, even if they are unable to remember the details of how or why the money was spent or obtained. The best explanation of this is that they obtained summary information for another purpose, such as tax records, or because they budgeted a specified amount of money for a vacation. If summary information is likely to be available from records and is all that is required, you should use that information instead of taking data for a much shorter time period and calculating the yearly amount. Ordinarily, however, you will be interested in both the summary data and the details of individual events. In this case, both summary questions and detailed questions for a short time period should be asked. Comparing the summary results with those obtained from extrapolating the data from the shorter period allows you to check the reliability of responses. How to Minimize Telescoping. An appropriate time period is also important if you are to minimize backward telescoping, or remembering events as happening more recently than they did. Suppose that a national sample of households are asked to report the amount of coffee they purchased in the past seven days and that this total is then compared with shipments of all coffee manufacturers or observed sales in retail outlets. These comparisons usually show that the amount reported is more than 50 percent higher than the amount manufactured and sold. What is happening is a process called telescoping. Telescoping results when the respondent remembers that the event occurred but forgets the exact date. In the past, most researchers were not concerned about telescoping because they believed that errors in the dates would be randomly distributed around the true date. However, recent research indicates that, as time passes, respondents are more uncertain about dates. As a result, respondents typically round their answers to conventional time periods, such as ten days ago, one month ago, or three months ago. The result of these two processes is to produce an overstatement of the reported events. Thus, an overstatement of coffee pur-
Slide 87: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 67 chasing occurs because respondents who bought coffee two or three weeks ago are likely to report that they purchased it in the last ten days or two weeks. Telescoping Biases Increase with Short Time Periods. Unlike simple omissions, which increase with the length of the time period, telescoping biases increase as the time period between the interview and the event is reduced. The worst problems with telescoping are for very short periods—yesterday, the last three days, last week. The reason is evident. Respondents who misremember by only a day will overreport by 100 percent if asked about yesterday, will overreport by about 7 percent if asked about the past two weeks, and will overreport by only 1 percent if asked about the last three months. For longer periods, absolute deviations from the correct date increase, but the relative deviations become smaller. If the behavior is highly salient, so that the percentage of omissions is small, substantial overstatements will occur if the time period is too short. In this case the researcher’s desire for a longer time period to obtain more data coincides with the selection of a time period to obtain the most accurate recall. Since both telescoping and omissions are occurring simultaneously, and since the effects of time work in the opposite directions for these two forms of forgetting, there is some time period at which the opposite biases cancel and the overall levels of reported behavior are about right. (See Sudman and Bradburn, 1974, for fuller discussion.) For many kinds of behavior—such as grocery shopping, leisure activities, and routine medical care—this period appears to be between two weeks and one month. Even when an optimum time period is selected, however, the details of the behavior still may not be correct for particular individuals. Use Bounded Recall Bounded-recall procedures, as developed by Neter and Waksberg (1963, 1964, 1965), involve repeated interviews with the same
Slide 88: 68 ASKING QUESTIONS respondents (a panel study). The initial interview is unbounded, and the data are not used for this period. At all subsequent interviews, however, the respondent is reminded of behaviors reported previously. The interviewer also checks new behaviors reported with those reported earlier, to make sure that no duplication has occurred. Thus, to prevent errors on dates, the earlier interviews “bound” the time period. Bounded interviews have been used successfully in a wide range of applications. Note, however, that the effects of bounding are just the opposite of those for aided recall. Bounding will reduce telescoping and improve information on details but will have no effect on omissions. If omissions are the more serious problem, bounded interviews may even cause larger errors, since bounded interviews eliminate compensating biases. Using both aided recall and bounded recall should result in low actual and net biases. The major problems with the bounding procedures now in use are that they require multiple interviews and may be too costly and too time-consuming for most researchers. An alternative is to use bounding procedures in a single interview (Sudman, Finn, and Lannom, 1984). That is, you start with questions about an earlier time period and use the data from that period to bound the reports of the current period. Thus, for example, in an interview conducted in the middle of June, a respondent might first be asked about clothing purchases during the month of May. Questions would then be asked about clothing purchases in June, with the May date used for bounding. Although there may be concerns of compounded biasing (suppose the estimate for May is too high), this method provides surprisingly accurate estimates. Consider the Use of Secondary Records Another method of reducing telescoping and improving information on details is to use household records, where available (Sudman and Bradburn, 1974). A search for records, as with a household inventory, is best accomplished in a face-to-face interview.
Slide 89: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 69 Although such a search is not impossible in a telephone interview, it is unnerving for both respondent and interviewer, since neither can see what the other is doing. If the search takes too long, it could have an adverse effect on the subsequent flow of the interview. An alternative that has been used with phone surveys is to mail the respondent a questionnaire in advance, indicating the types of records that will be useful to consult. Thus, the search is usually conducted before the interview. Where records are available, the interviewer should note whether or not they were used, since more accurate reporting will come from the respondents who used records. On mail and Web-based surveys, respondents can be asked to use records, but there is no strong motivation for them to do so, nor any way for the researcher to determine whether they did. Of the many kinds of records available, some of the most commonly used include the following: Bills for goods and services usually have the date of purchase of the product or the date the service was rendered, as well as the name of the supplier and other details. These can be used for studies of medical care, legal care, home repairs, and gasoline purchases, as well as for other expenditures. Insurance reimbursement forms provide information on medical and other insured costs. Checkbook records or canceled checks provide similar information to bills, except that the details are not available or are less precise. Titles and leases provide information on the characteristics of dwelling units and motor vehicles. Other financial records, such as insurance policies, bankbooks, and stock certificates, provide information about assets and savings. All records, especially those dealing with financial assets, are likely to be considered somewhat personal by respondents. Although it is good to encourage respondents to use these records, do
Slide 90: 70 ASKING QUESTIONS not insist if they are reluctant or if the records cannot be readily located. Learn to Use Diaries and Panel Research An alternative procedure that reduces reliance on recall and thus provides more accurate information about behavior is the use of diaries. The respondent or diary keeper is asked to record events immediately after they occur, or at least on the same day. Diaries have been used for a variety of topics, including consumer expenditures, food preparation, automobile use, television viewing, and health care (see Figure 2.4). These are all examples of frequent, nonsalient events that are difficult to recall accurately. Diaries have been used primarily in panel studies, where households or individuals report their behavior over time, thus making it possible to measure change at an individual level (Sudman and Wansink, 2002). However, some panel studies, such as most voting studies, use repeated interviews and not diaries; and some menu studies and Bureau of Labor Statistics Consumer Expenditure Surveys only use diaries to obtain reliable information for a single time period. Diaries are not used more often because they are costly and require survey techniques that are not familiar to many researchers. One reason for the increased costs is that diary keepers are usually compensated for their record-keeping activities, whereas respondents to most other studies are not. Also, to obtain the same level of cooperation as on other careful personal interviews, personal recruiting (face-to-face or by telephone) and extensive personal follow-up activities are required. As a result, the cost of data gathering is greater than for one-time interviews, although the cost per unit of information obtained is lower. Some researchers have used less expensive mail procedures for recruiting and collecting diaries, but cooperation with these procedures is much lower. (See additional discussion in Chapter Ten.) Three
Slide 91: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 71 findings from Sudman and Wansink (2002) that are relevant to this topic are as follows: 1. Ledger diaries (as in Figure 2.4), where events are entered by category, yield slightly more accurate information and are easier for the diary keeper to fill out and for the researcher to process than are journal diaries, where events are entered in the sequence they occur. The categories are helpful because different types of events require different details. Also, the headings act as reminders to record keepers of what is required. 2. Diaries should be kept relatively short—probably no longer than ten to twenty pages. Longer diaries with more items cause underreporting, particularly on items on the center pages of the diary. 3. Diary studies should ask for reports of several items rather than a single type of behavior or purchases of a single product. Otherwise, the record keeper will focus on this behavior and is likely to change this behavior. A diary study that asks only for reports of purchases of cereal is likely to lead, at least in the short run, to increased purchases and consumption of cereal. Even though they are costly, diaries should be seriously considered if you are attempting to obtain accurate, detailed information about frequent, low-salience behavior. Use the Right Words The general principle is simple: use words that everyone in the sample understands and that have only the meaning you intend. Writing questions that satisfy this principle is a difficult art that requires experience and judgment. You must expect to engage in a good deal
Slide 92: 72 ASKING QUESTIONS of trial and error—as well as pilot testing—before all the words are satisfactory. The obvious way to start is to use the simplest words that describe the behavior being measured. Often, however, many respondents may not know the single word that best describes the behavior. The classic, widely adopted solution suggested by Payne (1951) is to explain the word first and then provide the word itself. For example, the question “Do you procrastinate?” will confuse respondents who do not know what the word means, and the question “Do you procrastinate—that is, put off until tomorrow what you can do today?” may talk down to others. The best form of the question is to ask “Do you put off until tomorrow what you can do today—that is, procrastinate?” This form uses the technical word at the end and does not appear to talk down to respondents. Slang and colloquialisms should normally be avoided, not because such words violate good usage but because many respondents will not know what the words mean. If the sample is homogeneous and most respondents would use the same slang, however, the use of slang may be helpful. Thus, a study of delinquent boys from one ethnic group in a community could use the slang of that community. Here the use of slang is similar to the use of technical terms in a study of a professional group. In Figure 2.8, for example, the questions “Do you have a chisel plow?” and “Do you use reduced tillage?” are meaningful to most farmers although not to most college students, inner-city people, or survey researchers. When surveying unfamiliar groups, an initial group interview with a small (nonrandom) sample of that group may be helpful in indicating the types of words to use or avoid. Such group interviews are not definitive but may still be useful when the population studied is heterogeneous. Yet even more troublesome than an unknown word is a word that has multiple meanings in the context of the question being asked. Since many words have multiple meanings, we often depend on the context in which it is used to help clarify its meaning. Some words, however, are difficult to understand even in context. The
Slide 93: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 73 following examples are taken from Payne’s Rogue’s Gallery of Problem Words (1951, chap. 10): Any, anybody, anyone, anything. May mean “every,” “some,” or “only one.” Fair. Meanings include “average, pretty good, not bad,” “favorable, just, honest,” “according to the rules,” “plain,” “open.” Just. May mean “precisely,” “closely,” “barely.” Most. A problem if it precedes another adjective, as it is not clear whether it modifies the adjective or the noun, as in “most useful work.” Saw, see, seen. May mean “observe” or may mean “visit a doctor or lawyer.” Other words may have unexpected meanings to some respondents. A careful pilot test conducted by sensitive interviewers is the most direct way to discover these problem words. Since respondents’ answers may not always reveal their possible confusion about meanings, it is often useful to ask a respondent at the end of a pilot test “What did you think we meant when we asked [word or phrase]?” Determine the Appropriate Length of Questions It has generally been the practice to make questions as short as possible. This practice was based on research on attitude questions, which indicated that response reliability declines as the length of the question increases. Research on behavior questions, however, indicates that the findings for attitude questions do not apply to behavior questions (Cannell, Marquis, and Laurent, 1977; Cannell, Oksenberg, and Converse, 1977; Bradburn, Sudman, and Associates, 1979). For behavior topics, longer questions help reduce the number of omitted events and thus improve recall. There are three main reasons why longer questions improve recall.
Slide 94: 74 ASKING QUESTIONS First, longer questions provide memory cues and act as a form of aided recall. In an experiment, we compared a short question with a long question dealing with wine drinking. Note that the longer question lists possible uses of wine and reminds the respondents of possible settings and occasions to help with recall. Did you ever drink, even once, wine or champagne? (If yes): Have you drunk any wine or champagne in the past year? Wines have become increasingly popular in this country during the last few years. (By wines, we mean liqueurs, cordials, sherries, and similar drinks, as well as table wines, sparkling wines, and champagne.) Have you ever drunk, even once, wine or champagne? (If yes): You might have drunk wine to build your appetite before dinner, to accompany dinner, to celebrate some occasion, to enjoy a party, or for some other reason. Have you drunk any wine or champagne in the last year? The second reason longer questions can result in better accuracy is that the longer question takes more time for the interviewer to read, and it gives respondents more time to think. All else being equal, the more time respondents spend on the memory task, the more they will recall. The third reason longer questions are used has to do with the recent finding in psychological experiments that the length of the reply is directly related to the length of the question. If the interviewer talks more, the respondent will also talk more. If a written question is longer, the respondent will write more. Although length of response is not necessarily a direct measure of quality of response (particularly on attitudinal questions), longer responses will often lead to remembering additional events, cued by the respondent’s own conversation. Yet longer questions have the same possible disadvantages as aided recall. Although longer questions reduce omissions, the implicit demand for a positive response may increase telescoping.
Slide 95: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 75 Thus, as we shall see in the next chapter, long questions are useful for behavior that may be socially undesirable but may lead to an overreporting of socially desirable behavior. Using Respondents as Informants Up to this point we have mostly assumed that respondents are reporting only their personal behavior. For cost and availability reasons, you will often want respondents to report about other members of the household, and you may sometimes even want them to report about friends or organizations. Thus, one household informant, usually the principal shopper, may be asked to report about food purchases of all household members; a mother may be asked about the illnesses and doctor visits of all her children. Or one adult may be asked to report on the voting behavior of all other adults in the household. You would expect, and research confirms, that reports about others are generally 10 to 20 percent less accurate than reports about the respondent’s own behavior, unless that behavior is threatening (Marquis and Cannell, 1971; Menon, Bickart, Sudman, and Blair, 1995). Informants learn about the behavior of others by participating in the activity with them, observing them, or talking with them about it. In some cases, the informant may not know about the behavior. For example, children may purchase snacks away from home or participate in leisure activities that parents don’t know about. The behavior may also be unimportant and hence go unnoticed, such as purchasing personal care products or listening to the radio. In still other cases, the behavior, such as a minor illness, may not be salient and may even be forgotten by the person who was not directly involved. However, if the respondent does know about the salient behavior, such as a hospitalization or voting, information from informants may be highly reliable. The use of informants is especially efficient when you are screening the population for specific qualities, such as for people who golf or who are Gulf War veterans. False positives
Slide 96: 76 ASKING QUESTIONS can be eliminated in the more extensive interview conducted with the subsample of people who are reported to have the given characteristic. False negatives, those with the required attribute who are not reported by informants, are, however, missed by this screening. Summary In this chapter we have stressed that respondents may not be able to recall previous behavior. Anything that can be done to make this task easier should lead to improved quality of data as well as increased respondent and researcher satisfaction with the interview. The techniques suggested for helping jog the respondent’s memory of an event and for reducing telescoping included the following: (1) include aided recall, (2) make the question specific, (3) select an appropriate time period to ask about, (4) use bounded recall, (5) use secondary records, (6) use diaries and panel research, (7) use the right words, and (8) make questions the appropriate length. In general, it is critical to select tasks that can be realistically accomplished. Here the use of informants must be considered. With easy tasks and appropriate procedures, highly accurate reports of behavior can be obtained from these informants. When the task is difficult, however, even the best procedures will not produce error-free results. In this situation the best alternative, in our judgment, is neither to reject all results because complete accuracy cannot be obtained nor to ignore the basic problems with the data. Use the data with caution, recognizing their limitations but also recognizing that slightly flawed results can often be better than no results at all. Additional Reading The reader interested in the research findings that led to the recommendations in this chapter will find them in Thinking About Answers (Sudman, Bradburn, and Schwarz, 1996; see especially
Slide 97: ASKING NONTHREATENING QUESTIONS ABOUT BEHAVIOR 77 chaps. 6–9); Autobiographical Memory and the Validity of Retrospective Reports (Schwarz and Sudman, 1994); and our two earlier books, Response Effects in Surveys (Sudman and Bradburn, 1974; see especially chap. 3) and Improving Interview Method and Questionnaire Design (Bradburn, Sudman, and Associates, 1979, chap. 2). The research on counting and estimation is found in Blair and Burton (1987). For useful general books on memory, see Human Memory: Theory and Practice (Baddeley, 1990); Organization of Memory (Tulving and Donaldson, 1972); and Elements of Episodic Memory (Tulving, 1983). See also Linton (1975, 1978, 1982); Means and Loftus (1991); and Wagenaar (1986). For details on how to use diaries and to conduct and analyze research from either continuous or discontinuous consumer panels, see Consumer Panels, 2nd ed. (Sudman and Wansink, 2002). Chapters 9 and 10 in The Art of Asking Questions (Payne, 1951) are especially useful as supplementary reading for this chapter. The reader who wishes to become familiar with current research on questionnaire construction should consult the following journals, which frequently feature such research: Public Opinion Quarterly, Journal of Marketing Research, Journal of the American Statistical Association, Journal of Consumer Research, Journal of Personality and Social Psychology, Sociological Methods and Research, and Census Bureau Technical Papers.
Slide 99: Chapter Three Asking Threatening Questions About Behavior Survey researchers have long recognized that threatening questions about a person’s behavior need to be carefully worded. Barton (1958, p. 67) amusingly summarized many of the techniques nearly half a century ago. At the time, pollsters devoted time and energy to discovering ways to ask embarrassing questions in nonembarrassing ways. We give here examples of a number of these techniques, as applied to the question “Did you kill your wife?” a. The Casual Approach: “Do you happen to have murdered your wife?” b. The Numbered Card Approach: “Would you please read off the number on this card that corresponds to what became of your wife?” (Hand card to respondent) (1) Natural death (2) I killed her (3) Other (What?) c. The Everybody Approach: “As you know, many people have been killing their wives these days. Do you happen to have killed yours?” d. The “Other People” Approach: (1) “Do you know any people who have murdered their wives?” (2) “How about yourself?” 79
Slide 100: 80 ASKING QUESTIONS e. The Sealed Ballot Approach: In this version you explain that the survey respects people’s rights to anonymity in respect to their marital relations, and that they themselves are to fill out the answer to the question, seal it in an envelope, and drop it in a box conspicuously labeled “Sealed Ballot Box” carried by the interviewer. f. The Kinsey Approach: At the end of the interview, stare firmly into respondent’s eyes and ask in simple, clear-cut language, such as that to which the respondent is accustomed, and with an air of assuming that everyone has done everything, “Did you ever kill your wife?” Some of the basic procedures described by Barton are still used today, but others have been discarded as ineffective. In addition, the development of powerful computer technology has led to new methods that increase respondents’ confidence that their answers are confidential. Yet as questions become more threatening, respondents are more likely to overstate or understate behavior, even when the best question wording is used. For example, to this day, one of the most threatening questions still concerns household income. (Chapter Nine discusses how to reduce the threat of this question.) Checklist of Major Points 1. Self-administered computer-assisted procedures can reduce question threat and improve reporting on sensitive questions. 2. Open questions are generally better than closed questions for obtaining information on the frequencies of socially undesirable behavior. Closed questions, however, may reduce the threat of reporting whether or not one has ever engaged in a socially undesirable behavior. 3. Long questions are better than short questions for obtaining information on frequencies of socially undesirable behavior.
Slide 101: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 4. The use of familiar words may increase the frequency with which socially undesirable behaviors are reported. 5. To reduce overreporting of socially desirable behavior, use data from knowledgeable informants when possible. 6. For socially undesirable behavior, it is better to ask whether 81 the respondent has ever engaged in the behavior before asking whether they currently engage in that behavior. For socially desirable behavior, it is better to ask about current behavior first rather than asking about their usual or typical behavior. 7. To reduce the perceived importance of the threatening topic, embed it in a list of more and less threatening topics. 8. Consider alternatives to standard questions such as randomized response or card sorting. 9. Do not depend on wording such as “Did you happen to” to improve reporting of threatening questions. Such wording may actually increase the perceived threat of the question. 10. To increase both reliability and validity, consider using diaries or asking questions in several waves of a panel. 11. Avoid asking the same question twice in a questionnaire as a reliability check. This will annoy respondents and may increase the perceived importance of the topic to the respondent. 12. Ask questions at the end of the interview to determine how threatening the topics were perceived to be by the respondent. 13. Attempt to validate, even if only on an aggregate level. Six Examples of Questions on Socially Desirable Behavior Disease Detection Activities Health care researchers have usually found that disease prevention activities are seen as desirable behavior and are overreported when compared to medical records. Figure 3.1 gives a series of questions
Slide 102: 82 ASKING QUESTIONS asking about tests used to detect possible cancers in women. This series of questions asks about mammograms, Pap smears, and breast examinations by a physician. All preventive activities tend to be substantially overreported, but mammograms are more distinct events and are reported more accurately than Pap smears and breast examinations. Different study results vary in the degree of overreporting, but mammograms are roughly overreported by 40 percent, Pap smears and breast examinations by 100 percent or more. The questions attempt to sequence the behavior in a way that improves reporting accuracy. Library Card Ownership What is the best way to determine if a person owns a library card? It might seem as though a straightforward question would work best. Consider the question “Do you have a library card for the Denver Public Library in your own name?” On the surface, this question may appear nonthreatening, but since reading is generally considered a desirable activity, library card ownership is likely to be overstated. In fact, with this particular question, people overstated library ownership by 10 to 20 percent (Parry and Crossley, 1950). For socially desirable behavior, the extent of overstatement depends not only on the level of desirability and the wording of the question, but also on the proportion of the population who have not behaved in the socially desirable manner. Thus, the potential for overstatement is greater on library card ownership than for seat belt usage, since only a minority of adults have library cards. Figure 3.2 shows another approach that was used when trying to assess library card ownership in the Chicago area. This version attempted to reduce overreporting by asking additional questions about attitudes toward library facilities and about card ownership of other household members—thereby removing the stress from one specific question. Yet even in this version there was still 10 to 20 percent overreporting by all respondents. There
Slide 103: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 83 Figure 3.1. Questions on Cancer Screening. 1. a. A Pap smear is a routine test, often part of a pelvic examination, where the doctor uses a swab to take a sample from the mouth of the womb or cervix. Have you ever had a Pap smear test? Yes No b. How many Pap smear tests have you had in the past five years, since (Month) (Year)? Pap smear tests 2. a. A mammogram is an x-ray taken only of the breasts by a machine that presses the breast against a plate. Have you ever had a mammogram? Yes No b. How many mammograms have you had in the past five years, since (Month) (Year)? mammograms 3. a. A breast examination is when the breast is felt for lumps by a doctor or medical assistant. Have you ever had a breast examination by a doctor or medical assistant? Yes No b. How many breast examinations have you had in the past five years, since (Month) (Year)? breast examinations Source: Survey Research Laboratory, cited in Sudman and others, 1997. was no evidence that this version was effective. The same level of overreporting (or possibly a little larger) was found in Chicago as was found earlier in Denver. Book Reading Figure 3.3 shows various questions about book reading. As with library card ownership, book reading is a socially desirable activity, and the proportion of persons reading a book in the past six months
Slide 104: 84 ASKING QUESTIONS Figure 3.2. Questions on Library Card Ownership. 1. Would you say the Chicago Public Library facilities in your neighborhood are good, fair, or poor? Good Fair Poor Don’t Know 2. Does anyone in your family have a library card for the Chicago Public Library? Yes No (End Interview) 3. Do you have your own Chicago Public Library Card? Yes No Source: Bradburn, Sudman, and Associates, 1979. is probably overstated. The degree of overstatement is unknown, since no outside validating information is available. As Figure 3.3 indicates, one approach (National Opinion Research Center [NORC], 1965) to inquiring about readership is to ask “Have you read any book, either hard cover or paperback, within the past six months? (If you’ve started but not finished a book, that counts too.)” A second approach (NORC, 1963) would be to ask first about magazine reading and then about book reading. The idea here is that making book reading just one of several items about reading will reduce the focus on this item and the tendency to overreport. Question 1 is longer and provides the types of memory cues discussed in Chapter Two. The results, however, showed no difference in the proportion of people who read a book in the past six months. In both versions half of all respondents reported themselves to be readers. The wording in the third approach (Gallup, 1971) does not make the question specific (as we recommended in Chapter Two, since, instead of indicating a set time period it asks “When did you
Slide 105: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 85 Figure 3.3. Questions on Reading. APPROACH 1 Have you read any book, either hard cover or paperback, within the past six months? (If you’ve started but not finished a book, that counts too.) Yes No APPROACH 2 a. Do you read any magazines regularly? Yes No b. Have you read a book in the past six months? Yes No APPROACH 3 When, as nearly as you can recall, did you last read any kind of book all the way through either a hardcover book or a paper bound book? (If date is given) What was the title? [The Bible and textbooks were omitted.] Source: National Opinion Research Center, 1965. last read . . . ?”). It may, however, avoid the overstatements of socially desirable behavior that occur when a specific time period is indicated. The Gallup published results are based on reading in the past month, excluding Bible reading and textbooks, and thus cannot be directly compared with the National Opinion Research Center (NORC) results. Seat Belt Usage The Gallup (1973) wording of this question is “Thinking about the last time you got into a car, did you use a seat belt?” The question is careful not to ask about usual or typical behavior. To have asked about typical behavior would be more threatening and those who do not wear seat belts would be less likely to admit it. Asking only about a single event from among common events, such as getting
Slide 106: 86 ASKING QUESTIONS into a car, might appear to reduce the amount of information obtained, but many respondents will answer this question by accurately reporting what they usually do since it is difficult to remember separate episodes of routine behaviors. Charitable Giving The wording of a question in a study in charitable giving in Denver (Parry and Crossley, 1950), used to obtain information on contributions to a local charity (called the Community Chest), combined the specificity suggestions made in the previous chapter with the suggestion to cross-reference secondary sources. One survey question, for example, asks, “Did you, yourself, happen to contribute or pledge any money to the Community Chest during its campaign last fall?” The results were compared with the actual records of the Denver Community Chest. About one-third of the respondents reported giving and actually gave, and 34 percent did not give but reported that they did. As may be seen, words such as “happen to” evidently have little effect on reducing overreporting. In such a case, using a more direct question does not appear to have much effect on overreporting. Voting and Voter Registration Voting studies are among the most frequently conducted in the United States. Studies conducted after elections, however, almost always overstate the fraction of the population voting, as well as the fraction voting for the leading candidate. Figure 3.4 gives examples of voting and registration questions used by several survey organizations at various times during the course of a year. Despite their attempts to reduce overreporting, most were unsuccessful. A number of different strategies are used to attempt to reduce overreporting. Some of these strategies are as follows:
Slide 107: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 87 Figure 3.4. Voting Questions. 1. In 1996 Bill Clinton ran on the Democratic ticket against Bob Dole for the Republicans, and Ross Perot as an independent candidate. Do you remember for sure whether or not you voted in that election? Yes, Voted No, Didn’t Vote Don’t Know (If Yes) Which one did you vote for? Bill Clinton Bob Dole Ross Perot Other (Specify) Don’t Know 2. In talking to people about elections, we often find that a lot of people were not able to vote because they weren’t registered, they were sick, or they just didn’t have time. Which of the following statements best describes you: One, I did not vote (in the election this November); Two, I thought about voting this time—but didn’t; Three, I usually vote, but didn’t this time; or Four, I am sure I voted? I did not vote (in the election this November) I thought about voting this time, but didn’t I usually vote, but didn’t this time I am sure I voted Source: Burns and others, 2001. Using words like “for certain” or “for sure” to indicate that not remembering is a possible answer Indicating that there are good reasons why people cannot always vote Providing the names of the candidates as a reminder and to help avoid confusion with other elections Asking the question as one of a series of other questions dealing with political attitudes Interestingly, one method that effectively reduces overreporting is to obtain voting information from a household informant for all household members, instead of obtaining individual data. The
Slide 108: 88 ASKING QUESTIONS relationship between household informants and reporting accuracy is an important one that will be discussed later on in this book. Reporting accuracy is also a function of the actual percentage of people voting. Presidential election questions are probably answered most accurately, since more registered voters actually vote in presidential elections. In May 1949, 13 percent of the population claimed to have voted in the 1948 presidential election and did not, according to the voting records. People were also asked if they had voted in the presidential election four years earlier in 1944. Of those claiming to have voted, 23 percent could not be matched with reported voting records. Of respondents in the NORC survey, 65 percent reported that they had voted in 1976 (General Social Survey, 1977–78). The Current Population Survey based on data from a household informant indicated that 59 percent of the population had voted (U.S. Bureau of the Census, 1976). The actual percentage voting was 57.5 percent. Although some of the differences between NORC and the Current Population Survey may be the result of other factors, these results suggest that for voting and other socially desirable behaviors, more reliable information is obtained from an informant rather than from the individual directly, because informants find questions about others less threatening than questions about themselves. Substantially higher overreporting of voting is possible in elections where fewer people vote. Primary and most local elections are less salient to respondents and easier to confuse with other elections. It is also difficult for the researcher to provide memory cues, because the list of candidates is long. For these elections, it would be especially desirable to use an informant if possible. Nevertheless, substantial overstatements should still be expected. Ideas about what technique best alleviates errors in reporting on voting have changed in recent years. In a recent survey regarding November 2000 voting and registration, the U.S. Census Bureau focused on increasing the number of people who truthfully completed the questionnaire. (See Figure 3.5.) The questions therefore
Slide 109: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 89 Figure 3.5. Voting Questions that Improved Accuracy. 1. In any election, some people are not able to vote because they are sick or busy or have some other reason, and others do not want to vote. Did you vote in the election held on Tuesday, November 7? Yes No 2. What was the main reason you did not vote? Illness or disability (own or family’s) Out of town or away from home Forgot to vote (or send in absentee ballot) Not interested, felt my vote wouldn’t make a difference Too busy, conflicting work or school schedule Transportation problems Didn’t like candidates or campaign issues Registration problems (i.e. didn’t receive absentee ballot, not registered in current location) Bad weather conditions Inconvenient hours, polling place or hours or lines too long Other Source: U.S. Bureau of the Census, 2001. centered on making participants feel comfortable reporting on whether or not they voted in the presidential election. They were first asked if they voted. The initial question took great care to make it clear that a person may not be able to vote or may choose not to vote for many reasons. If respondents answered no, they were then asked why they did not vote, and they were given eleven different answer options from which to choose. The next questions then dealt with when and how the person registered to vote. The intent of this survey was clearly to obtain an accurate estimate of the number of voters in the 2000 election. In contrast to previous U.S. Census Bureau reports, this report based voting and registration rates on the citizen population of voting age—taking into account that not all people of voting age in the United States are citizens and are eligible to vote. This change in criteria also changed the end voter turnout rate a full five percentage points,
Slide 110: 90 ASKING QUESTIONS from 55 percent to 60 percent of voting-age citizens. In the end, though, this is only one attempt in a line of attempts to garner a more accurate response from the survey. Four Examples of Questions on Socially Undesirable Behavior Traffic Violations Traffic violations range in threat from relatively minor violations, such as a parking ticket, to much more serious violations, such as driving under the influence of alcohol. In a methodological study we conducted in Chicago, two separate strategies were used in an attempt to improve reporting. First, a series of questions was asked about various traffic violations so that respondents would not know that driving under the influence of alcohol was the topic of primary research interest. Second, randomized response procedures were used. Figure 3.6 shows how the interviewer presented these procedures. (These procedures were also used on other topics in the questionnaire.) Using randomized response makes it possible to estimate the proportion of the population engaging in certain threatening behaviors, but it does not help determine the isolated behavior of a single, specific respondent. (The procedure is explained in greater detail later in the chapter.) Randomized response is one method for assuring respondents that the interviewer will not know what answer is given. It gives them a feeling of anonymity. Another method is to use selfadministered questionnaires. Although increasing the anonymity of response generally reduces reporting errors, such errors are not eliminated entirely. (See Figure 3.6.) For very threatening socially undesirable behavior such as drunk driving, 35 percent of a sample of respondents chosen from traffic court records denied being charged with drunken driving even with randomized response procedures. Still, this was a lower response error than the 50 percent underreporting that was found using standard procedures.
Slide 111: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 91 Figure 3.6. Survey of Traffic Violations. 1. There are some questions that are asked in survey research that are too difficult to ask directly because many people think they are too personal. While it is understandable that people feel this way, there is a real need for information for the population as a whole. We now have a way that makes it possible for people to give information, without telling anyone about their own situation. Let me show you how this works: we will use the next question I have here as an example. (Hand R. Card F.) As you see, there are two questions on the card. One deals with the “real” question that the research is concerned with; the other is completely unrelated. Both questions can be answered “yes” or “no.” One of the two questions is selected by chance and you answer it. (I’ll show you how that works in a minute.) I do not know which questions you are answering. . . . (Hand R. box.) It is very simple, as you will see. You use this little plastic box. Notice that the box has red and blue beads in it. By shaking the box, you can get one of those beads to show in the little “window” in the bottom corner of the box. Try it. (Encourage R. to “play with” the box a little, to get used to it.) Okay. Now you’ll notice that one of the questions on the card has a red circle next to it, and one has a blue circle. The question that you answer is selected by chance. Shake the box again and look at the color of the bead that shows in the window now—don’t tell me what color it is. If the bead is blue, you answer the “blue circle” question on the card; if the bead is red, you answer the “red circle” question. I can’t see which bead is in the window; and you don’t tell me which question you are answering. Just tell me if your answer is “yes” or “no.” CARD F. (Red) Have you received a ticket for parking in the last twelve months? (Blue) Is your birthday in the month of June? 2. (Hand R. Card G.) Please shake the box again and, using this card, answer the question whose color matches the bead in the window. Is your answer “yes” or “no”? CARD G. (Red) Have you received a ticket for going through a red light in the past 12 months? (Blue) Is your birthday in the month of July? 3. (Hand R. Card H.) Now shake the box again and, using this card, answer the question whose color matches the bead in the window. Is your answer “yes” or “no”? CARD H. (Red) During the last 12 months, have you been charged by a policeman for speeding? (Blue) Is your birthday in the month of August? 4. (Hand R. Card I.) Now shake the box again. Use this card and answer the question whose color matches the bead in the window. Is your answer “yes” or “no”? CARD I. (Red) During the last 12 months, have you been charged by a policeman for driving under the influence of liquor? (Blue) Is your birthday in the month of September? Source: National Opinion Research Center, 1972, cited in Bradburn, Sudman, and Associates, 1979.
Slide 112: 92 ASKING QUESTIONS Illegal Drug Use Illegal drug use is a major health as well as a major legal problem, and measuring trends is critical for policy purposes. It is obvious that illegal drug use is seen as a socially undesirable behavior among most segments of society. In annual studies conducted by the Research Triangle Institute (RTI) for the National Institute on Drug Abuse, self-administered procedures have been developed so that the interviewer does not know what answers were given. Figure 3.7 gives excerpts from the questionnaire that indicate the information requested. The most direct method for a self-administered questionnaire would be to have respondents read the questions and answer them with no interviewer present. Unfortunately, the drug questionnaire is complex and many respondents would have trouble reading the questions and following the instructions. Initially, RTI attempted to solve this problem by having the interviewer read the questions while the respondent filled out an answer sheet. More recently they have used an audio computer-assisted self interview. With this method, which will be discussed in greater detail later in this chapter, the respondent is given a laptop computer and earphones. The questions are asked by a recorded voice and the respondent enters the answers into the computer. Again it should be stressed that such self-administered methods reduce, but do not eliminate all reporting errors. It might appear that a question like number 3 in Figure 3.7 asking for number of days of lifetime use would be difficult or impossible for users to answer. One possible value for such a question pointed out by Wentland and Smith (1993) is that such a question may suggest by the answer categories that the use of marijuana is widespread, which would reduce underreporting of marijuana use. Use of Alcoholic Beverages Although many people do not find questions about their use of alcoholic beverages to be threatening, some—especially heavy
Slide 113: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 93 Figure 3.7. Questions on Illegal Drug Use. 1. The next questions are about marijuana and hashish. Marijuana is also called “pot,” “grass,” or “dope.” Marijuana is usually smoked, either in cigarettes called joints, or in a pipe. It is sometimes cooked in food. Hashish is a form of marijuana that is also called “hash.” It is usually smoked in a pipe. Have you ever, even once, used marijuana or hashish? Yes No 2. How old were you the very first time you actually used marijuana or hashish? years old I have never used marijuana or hashish 3. Altogether, on how many days in your life have you used marijuana or hashish? I have never used marijuana or hashish 1 to 2 days 3 to 5 days 6 to 10 days 11 to 49 days 50 to 99 days 100 to 199 days 200 to 299 days 300 to 399 days 400 or more days 4. How long has it been since you last used marijuana or hashish? I have never used marijuana or hashish Within the past 30 days More than 30 days but less than 6 months ago 6 months or more but less than 1 year ago 1 year or more but less than 3 years ago 3 years or more ago 5. Think specifically about the past 30 days—that is, from your 30 day reference date up to and including today. During the past 30 days, on how many days did you use marijuana or hashish? number of days on which I used marijuana or hashish I have used marijuana or hashish, but not in the past 30 days I have never used marijuana or hashish (Similar questions asked about cigarettes, alcohol, analgesics, tranquilizers, stimulants, sedatives, inhalants, hallucinogens, cocaine and heroin.) Source: Research Triangle Institute, 1990, cited in Turner, Lessler, and Gfroerer, 1992.
Slide 114: 94 ASKING QUESTIONS drinkers—may. Figure 3.8 presents a series of questions, the first asked by Gallup to determine whether respondents use alcohol, how much respondents use alcohol, and how often. The Gallup question is short, simple, and explicit. The NORC questions formed a series in a longer questionnaire and were designed to make it easier for respondents to recall and to admit their use of alcoholic beverages. The progressive nature of the questioning makes it easier for heavy drinkers to answer the questions without avoiding the issue or underreporting their consumption frequency. Note that respondents are first asked whether they have ever drunk alcoholic beverages or a specific beverage such as beer. At this stage some respondents who have used and are using alcohol will not want to admit it. If they deny alcohol use, they will not be asked any questions about current behavior. A comparison of the Gallup and NORC versions suggests that the NORC version—which asked if respondents used beer in one question, or wine in another question, or other specific alcoholic beverages in other questions—is perceived as less threatening than the Gallup version, which asked if respondents used any alcoholic beverage. A higher percentage of respondents report using wine and beer in the NORC version than report using any alcoholic beverage in the Gallup question. This is probably because some respondents who are willing to report using wine and beer may be unwilling to report using alcoholic beverages because this term is perceived as meaning hard liquor. After respondents report that they have drunk beer, wine, or liquor, it is effective to use long and open-ended questions to ask respondents how much they drink—such as “When you drank beer, how often did you drink it on average?”—without giving any indication of a possible range of answers. When we compared these open-ended questions with short closed-ended questions where answer categories were given, we found that the quantities reported on the long, open-ended form were more than double those on the
Slide 115: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 95 Figure 3.8. Questions on Alcoholic Beverage Use. 1. Do you ever have occasion to use any alcoholic beverages such as liquor, wine, or beer, or are you a total abstainer? 2. Now we have some questions about drinking for relaxation. The most popular alcoholic beverage in the country is beer or ale. People drink beer in taverns, with meals, in pizza parlors, at sporting events, at home while watching television, and many other places. Did you ever drink, even once, beer or ale? (If no, go to Q. 3) (If yes) We are especially interested in recent times. Have you drunk any beer or ale in the past year? Yes (Ask a, b, and c.) No (Ask a and b.) a. When you drank beer or ale, on the average how often did you drink it? Include every time you drank it, no matter how little you had. b. Most of the times you drank beer, on the average how many bottles, cans, or glasses did you drink at one time? c. Thinking about more recent times, have you drunk any beer or ale in the past month? 3. Wines have become increasingly popular in this country over the last few years; by wines, we mean liqueurs, cordials, sherries, and similar drinks, as well as table wines, sparkling wines, and champagne. Did you ever drink, even once, wine or champagne? (If no, go to Q. 4.) (If yes) You might have drunk wine to build your appetite before dinner, to accompany dinner, to celebrate some occasion, to enjoy a party, or for some other reason. Have you drunk any wine or champagne in the past year? Yes (Ask a, b, and c.) No (Ask a and b.) a. When you drank wine or champagne, on the average how often did you drink it? Include every time you drank it, no matter how little you had. b. Most of the times you drank wine or champagne, on the average about how many glasses did you drink at one time? c. Thinking about more recent times than the past year, have you drunk any wine or champagne in the past month? 4. Part of our research is to try to find out the best way to ask questions. Sometimes, as we go through the questionnaire, I’ll ask you to suggest terms that we might use, so that you will feel comfortable and understand what we mean. For instance, my next few questions are about all the drinks like whiskey, vodka, and gin. What do you think would be the best things to call all the beverages of that kind when we ask questions about them? (If no response, or awkward phrase, use “liquor” in following questions. Otherwise, use respondent’s word(s).)
Slide 116: 96 ASKING QUESTIONS Figure 3.8. Questions on Alcoholic Beverage Use, continued. People drink __________ by itself or with mixers such as water, soft drinks, juices, and liqueurs. Did you ever drink, even once, __________? (If no, go to Q. 5.) (If yes) You might have drunk __________ as a cocktail, appetizer, to relax in a bar, to celebrate some occasion, to enjoy a party, or for some other reason. Have you drunk any __________ in the past year? Yes (Ask a, b, and c.) No (Ask a and b.) a. When you drank __________, on the average how often did you drink it? Include every time you drank it, no matter how little you had. b. Most of the times you drank __________, on the average about how many drinks did you have at one time? c. Thinking about more recent times than the past year, have you drunk any __________ in the past month? 5. Sometimes people drink a little too much beer, wine, or whiskey so that they act different from usual. What word do you think we should use to describe people when they get that way, so that you will know what we mean and feel comfortable talking about it? (If no response, or awkward phrase, use “intoxicated” in following questions. Otherwise use respondent’s word(s).) (If R. has answered yes for drinking any alcohol in the past year) Occasionally, people drink a little too much and become __________. In the past year, how often did you become __________ while drinking any kind of alcoholic beverage? 6. Next, think of your three closest friends. Don’t mention their names, just get them in mind. As far as you know, how many of them have been __________ during the past year? Source: Q.1, Gallup, 1977; Q.2–Q.6, National Opinion Research Center, 1972, cited in Bradburn, Sudman, and Associates, 1979.
Slide 117: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 97 short, closed-ended form. In Questions 4 and 5, an additional effort was made to reduce the threat by asking respondents to use their own words when discussing liquor and drunkenness. This procedure also appeared to improve reporting, but not so much as the use of long, open-ended questions. Sexual Activity Figure 3.9 presents a series of self-administered questions from the NORC study on sexuality (Laumann, Gagnon, Michael, and Michaels, 1994). The researchers examined responses controlling for interviewer evaluations of respondents’ frankness and comprehension of the questions and saw no differences. Of course, this does not mean that all sexual activities were accurately reported. It has frequently been noted that interviewers cannot really tell when respondents are not answering truthfully. Nevertheless, the use of self-administered forms for these really threatening questions almost certainly reduced threat. Note that the frequency questions give answer categories rather than simply asking for an open answer on how many times or with how many partners. Respondents who found the question sensitive or had trouble remembering might have used a middle answer to indicate what researchers thought was the most likely possibility. Nevertheless, by far the most common answer given for the number of sexual partners was one. Again, it can be seen that the careful sequencing of these questions provides two advantages in improving reporting accuracy. First, it differentiates and clarifies the activities being investigated, so as to not cause confusion over the “definition” of sex. Second, it also leads up to the more uncomfortable questions by asking the more general questions to begin with.
Slide 118: 98 ASKING QUESTIONS Figure 3.9. Questions on Sexual Activity. 1. There is a great deal of concern today about the AIDS epidemic and how to deal with it. Because of the grave nature of the problem, we are going to ask you some personal questions and we need your frank and honest responses. Your answers are confidential and will be used only for statistical reports. How many sex partners have you had in the last 12 months? No partners 1 partner 2 partners 3 partners 4 partners 5-10 partners 11-20 partners 21-100 partners More than 100 partners 2. Have your sex partners in the last 12 months been . . . Exclusively male? Both male and female? Exclusively female? 3. About how often did you have sex in the past 12 months? Not at all Once or twice About once a month Two or three times a month About once a week Two or three times a week Four or more times a week 4. Masturbation is a very common practice. In order to understand the full range of sexual behavior, we need to know the answers to a few questions about your experiences with masturbation. By masturbation we mean self-sex or self-stimulation, that is, stimulating your genitals (sex organs) to the point of arousal, but not necessarily to orgasm or climax. The following questions are not about activity with a sexual partner, but about times when you were alone or when other people were not aware of what you were doing. On average, in the past 12 months how often did you masturbate? More than once a day Every day Several times a week Once a week
Slide 119: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 2-3 times a month Once a month Every other month 3-5 times a year 1-2 times a year 0 times this year FEMALES ONLY (A comparable set was asked of males) 5. Have you ever performed oral sex on a man? Yes No 6. Has a man ever performed oral sex on you? Yes No 7. Have you ever had anal sex? Yes No 8. Have you ever been paid by a man to have sex? Yes No Source: Laumann, Gagnon, Michael, and Michaels, 1994. 99 Nine Techniques to Make Threatening Questions More Accurate Use Self-Administered Methods Probably the most widely used method for reducing question threat is to make the question self-administered. This works for both socially desirable and socially undesirable behaviors. Respondents are less likely to overreport desirable behaviors such as voting or giving to charity when responding on a computer or piece of paper rather than answering an interviewer. They are also less likely to underreport socially undesirable behavior and even attitudes that are perceived as socially undesirable.
Slide 120: 100 ASKING QUESTIONS It must be recognized, however, that self-administered methods do not yield perfect information. Some behaviors are so threatening to respondents that they would not report the behavior even if assured of perfect anonymity. More often, however, respondents do not completely trust the assurances of anonymity. They recognize that someone, if not the interviewer, will be looking at their answers and that there is some possibility they might be personally identified. The more threatening the question, the less effective anonymity will be. Before the days of computer-assisted interviewing, the methods usually involved respondents reading the questions, circling the answer, and then putting the completed form in a sealed envelope that was returned to the interviewer. Alternatively, they would sometimes put completed forms into a sealed ballot box with a lock to simulate the box found at polling places. One especially effective form of self-administration involves group administration, whereby groups of people each complete individual questionnaires that are then collected into a common box or envelope. Since the forms contain no identifying information, group members have a strong feeling of anonymity. This is a likely explanation for why drug studies conducted in classrooms generally yield higher reports of usage than individual drug interviews. Currently, when the interview is conducted with a computer, the respondent is given a computer and enters the answers directly into it. If the respondent has difficulty reading, the interviewer may read the question and the respondent may enter the answer. Even in this scenario, having the interviewer read the question appears to reduce the respondent’s sense of anonymity. Audio computer methods that eliminate the interviewer entirely are now increasingly used. The questions are read by a recorded voice and the respondent hears them on a set of earphones and enters the answer. No one else in the room can hear the questions. Almost all the other features of computer-assisted interviewing discussed in detail in Chapter Ten (such as branching and skipping questions) are also available.
Slide 121: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 101 Use Card Sorting and Randomized Response Card sorting is a procedure that has been used in Great Britain in face-to-face interviews to measure crime and juvenile delinquency (Belson, Millerson, and Didcott, 1968). Here the interviewer hands respondents a set of cards that list various behaviors, including threatening ones. Respondents are asked to place each card into a “yes” or “no” box. During later points in the interview, the interviewer can ask the respondent to reconsider the cards in the “no” box and to resort the cards if necessary. The thought behind card sorts is that it might be easier for some respondents to admit a socially undesirable behavior (or not to claim a socially desirable behavior) when performing a nonverbal task. As far as we know, however, card sorting has not been empirically validated or compared with alternative procedures. The randomized response technique is really a randomized questioning technique. It is a method that ensures respondents’ anonymity by making it impossible for either the interviewer or researcher to know what question the respondent was answering (Greenberg and others, 1969; Horvitz, Shaw, and Simmons, 1967; Warner, 1965). Specifically, the interviewer asks two questions, one threatening and the other completely innocuous. For example, Question A contains a very threatening question, such as “During the last 12 months have you been charged by a policeman for driving under the influence of liquor?” Question B is a nonthreatening question, such as “Is your birthday in the month of September?” Both of these questions have the same possible answers, “yes” and “no.” Which question the respondent answers is determined by a probability mechanism. We and others have used a plastic box containing fifty beads, 70 percent red and 30 percent blue. The box was designed so that, when it was shaken by the respondent, a red or blue bead seen only by the respondent would appear in the window of the box. If the bead is red, the threatening question is answered; if blue, the innocuous question is answered.
Slide 122: 102 ASKING QUESTIONS To illustrate how the procedure works, suppose that out of a sample of 1,000 respondents, 200 answered “yes” to the pair of questions given above and 800 answered “no.” The expected number of persons answering “yes” to the question about the month of their birthday is approximately 25 (1,000 × .3 ÷ 12). This assumes that birth dates are equally distributed over the twelve months and that .3 of the respondents saw a blue bead. Thus, the net number of people answering “yes” to the question on drunken driving is 200 − 25, or 175. The number of people who answered Question A is approximately .7 × 1,000, or 700. The percentage of people who admit being arrested for drunken driving is 175 ÷ 700, or 25 percent. By using this procedure, you can estimate the undesirable behavior of a group while fully protecting the anonymity of the respondent. With this method, however, you cannot relate individual characteristics of respondents to individual behavior. That is, standard regression procedures are not possible at an individual level. If you have a very large sample, group characteristics can be related to the estimates obtained from randomized response. For example, you could look at all the answers of young women and compare them to the answers of men and older age groups. On the whole, much information is lost when randomized response is used. Even if the information obtained from randomized response were error-free (and it is not), the loss of information has made this procedure much less popular than it was when first introduced. The accuracy of information obtained by randomized response depends on the respondent’s willingness to follow instructions, understand the procedure, and tell the truth in exchange for anonymity. Unfortunately, for very threatening questions, such as our example of drunken driving given earlier, there is still substantial underreporting of socially undesirable behavior. Randomized response is also not an appropriate procedure for asking questions about socially desirable behavior, where it may lead to even higher levels of overreporting than standard methods. Randomized and anonymous response procedures are appropriate
Slide 123: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 103 to use when researching behavior such as abortion and bankruptcies, where the respondent may not personally be ashamed of the action but may not know how the behavior is viewed by the interviewer. Aside from the issue of reporting quality, some readers may wonder whether procedures such as randomized response and card sorting have any negative effects on respondent cooperation by disrupting the flow of the interview. All evidence indicates that quite the contrary is the case. Both respondents and interviewers enjoy card sorting exercises or shaking a box of beads. Interviewers report that respondent cooperation improves when there is some variety in the tasks. Use Open-Ended Questions As a general rule survey researchers prefer closed questions because they are easier to process and they reduce coder variability. (See Chapter Five.) In attempting to obtain frequencies of threatening behavior, however, there is no difficulty in coding, since the answer is numeric. For example, Question 2A in Figure 3.8 asks how often the respondent drank beer and allows for such answers as “Daily,” “Several times a week,” “Weekly,” “Monthly,” and so on. All these answers can be converted to number of days per month or year. It may not be obvious why the open-ended question here is superior to a closed-ended question that puts possible alternatives on a card and asks the respondent to select one. One reason is that the closed question must arrange the alternatives in a logical sequence, from most frequent to least frequent, or the reverse. In either case, the most extreme answer, “Daily,” would be either at the extreme top or bottom of a list provided on a card. Heavy drinkers who drank beer daily would need to select the extreme response if they reported correctly. There is, however, a general tendency for respondents to avoid extreme answers and to prefer an answer in the middle of a list because it is thought to indicate those values that the researcher thinks are most likely in the population.
Slide 124: 104 ASKING QUESTIONS This applies to attitude and knowledge questions as well as behavior. Thus, some of the daily drinkers would choose a response more in the middle, thereby causing a substantial understatement. An alternative explanation is that the open-ended questions allow the really heavy drinkers to state numbers that exceed the highest precodes. When researchers set precodes, they tend to set the highest value at a level that will still have fairly high frequencies. If the tail of a distribution is long, the highest precode category does not capture the really heavy drinkers. For more discussion of open questions, see Bradburn, Sudman, and Associates, 1979, Chapter Two. There is one occasion when asking a closed-ended question may be desirable. This is when one is interested in learning whether the respondent has ever done what some might consider a socially undesirable act, such as masturbation, in the past month. Asking a closed frequency question such as Question 4 in Figure 3.9 may suggest to respondents that masturbation is widely practiced and that an answer at the lower end of the scale would not shock anyone. Use Long Questions with Familiar Words The advantages and possible disadvantages of longer questions about nonthreatening behavior were discussed in Chapter Two, and that discussion need not be repeated. When questions are asked about the frequency of socially undesirable behavior, overreporting is not a problem with most segments, and longer questions help relieve the tendency to underreport. It is important to use these longer questions to try to provide additional cues to memory. Thus, Question 3 in Figure 3.8 begins by pointing out the popularity of beer and wine and listing examples of their uses. Longer questions increased the reported activities of socially undesirable behavior by about 25 to 30 percent, as compared with the standard short questions. Longer questions, however, had no effect on respondents’ willingness to report ever engaging in socially unde-
Slide 125: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 105 sirable activity, such as drinking liquor or getting drunk (Bradburn, Sudman, and Associates, 1979). Some critics of survey research procedures claim that using standardized wordings makes the interview situation artificial. They claim that such stilted wording makes it more difficult for some respondents to understand the question and provide truthful responses and that slang would be easier for most respondents to understand. Furthermore, these people contend that slang and colloquialisms are often used in normal conversation when the behavior being discussed is socially undesirable and thus are appropriate for questions regarding such behavior. In contrast to this position, other researchers are concerned that using slang and varying the question wording from respondent to respondent introduces uncontrolled method variability. This problem is greatest with attitude questions where answers clearly depend on how the question is asked, but less critical for behavior questions where understanding the question is most important. Still, how the behavior question is asked may significantly affect how threatening it is perceived to be. One approach that reduces the threat is to have the respondent (not the interviewer) make the decision on the word to use, when the standard words such as “liquor” or “sexual intercourse” may be too formal. In our research, we learned that most respondents preferred the term “love making” to “sexual intercourse,” and some used even more direct colloquialisms. For liquor, many respondents used words such as “booze.” The interviewer would then use the respondents’ words. This is easy to do in computer-assisted interviewing where once the word is typed into the computer by the interviewer it can be programmed to appear in subsequent questions. For example, Question 4A in Figure 3.8 would ask “When you drank booze, on the average how often did you drink it?” The use of familiar words increased the reported frequencies of socially undesirable behavior about 15 percent, as compared to the use of standard wording (Bradburn, Sudman, and Associates, 1979).
Slide 126: 106 ASKING QUESTIONS When some respondents are asked to give the word they would prefer to use, they either do not know or they give an inappropriate response. Thus, on the pretest one respondent used the word “poison” to describe liquor. In this situation the interviewer must always have a fallback word that he or she could type in. Typically, this is the standard word, such as “liquor” or “sexual intercourse.” Use Informants In the previous chapter, we pointed out the cost efficiencies of using household informants, but indicated that this might be at the cost of some loss in quality of information. For threatening questions, however, especially those dealing with socially desirable behavior, informants may provide more reliable information than respondents. It is, of course, necessary to ask about behavior that the informant might know about others, either from observation or through conversations. This could include topics such as voting, book reading, or use of alcohol and drugs (Bradburn, Sudman, and Associates, 1979). The question may be asked about identifiable members in the same household or about unidentified friends or relatives. In either situation, respondents will not be so threatened answering questions about the behavior of others as they would be answering questions about their own behavior. An exception to this rule is asking parents to report about children. Parents may be more threatened and thus report lower levels of socially undesirable behavior than their children, or they may just not know. Use Diaries and Panels In Chapter Two we discussed using diaries that provide repeated written records for improving memory about nonsalient events. Diaries and consumer panels also reduce the respondent’s level of threat. First, any event becomes less threatening if it is repeated
Slide 127: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 107 over time and becomes routine. Respondents who might initially hesitate to report purchasing beer or contraceptives become less inhibited as time goes by. Second, with repeated exposure, respondents gradually gain confidence in the organization or researcher gathering the data. Over time, respondents get a better understanding that the data are gathered to be used in aggregate form and that there are no personal consequences of reporting any kinds of behavior. The evidence suggests that confidence in the research and the perceived threat of the questions both level off fairly rapidly after two or three diaries or interviews. This is fortunate, since otherwise substantive data on trends would be confounded with response effects (Ferber, 1966). Finally, diaries embed some threatening topics into a more general framework to avoid conditioning. Such embedding also appears to be effective for reducing threat. For example, respondents who reported health expenditures in a diary (Sudman and Lannom, 1980) reported higher levels of expenditures for contraceptives than did respondents who were interviewed several times. The diaries here seem to be having the same effect as anonymous forms. Embed the Question The threat of a question is partially determined by the context in which it is asked. If more threatening topics have been asked about earlier, a particular question may appear less threatening than if it had been asked first. Yet there are limitations to the use of this procedure. As we shall see in Chapter Ten, you would not want to start with very threatening questions since this could reduce respondent cooperation during the rest of the questionnaire. Also, putting the most threatening behavior question first will probably make the underreporting on that question even worse. Suppose, however, you were interested only in beer drinking. Then you would ask an early question about liquor drinking to reduce the threat of the beer
Slide 128: 108 ASKING QUESTIONS drinking question. If you were particularly interested in shoplifting, you might use the following order (adapted from Clark and Tifft, 1966). Did you ever, even once, do any of the following: Commit armed robbery? Yes No Yes No Break into a home, store, or building? Take a car for a ride without the owner’s knowledge? Yes No Take something from a store without paying for it? Yes No In a more general sense, the threat of individual questions is also determined by the general context of the questionnaire. Thus, a questionnaire that deals with attitudes toward alcoholism is more threatening than a questionnaire that deals with consumer expenditures. Consequently, respondents may be more willing to admit that they use alcohol when the question is one of a series of questions about consumer expenditures or leisure-time activities or lifestyles. It can sometimes be difficult for a researcher to decide whether to use questions that are not directly related to the threatening topics being studied but are included only to embed the threatening questions. These added questions increase the length of the questionnaire and the cost of the study. We suggest, however, that judicious use of such questions can increase respondent cooperation and data quality with only small increases in cost. An artful investigator, when faced with the need to embed threatening questions, can choose additional questions that contribute to the richness of the research, even if these questions are not of primary interest. In other words, you can use other questions to take some of the emphasis off of your primary interest in a given survey.
Slide 129: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 109 Choose the Appropriate Time Frame All else equal, questions about events that have occurred in the past should be less salient and less threatening than questions about current behavior. Thus, for socially undesirable behavior, it is better to start with a question that asks “Did you ever, even once . . . ,” rather than to ask immediately about current behavior. Refer to Questions 2, 3, and 4 (about drinking beer, wine, and liquor) in Figure 3.8. Other examples might be the following questions about delinquent behavior. “Did you ever, even once, stay away from school without your parents knowing about it?” “Did you ever, even once, take something from a store without paying for it?” After asking “Did you ever . . . ,” the interviewer then asks about behavior in some defined period, such as the past year. As was pointed out in the previous chapter, it is difficult for respondents to remember accurately details on events in the distant past unless the events are highly salient. For socially desirable behavior, however, just the reverse strategy should be adopted. It would be very threatening for respondents to admit that they never did something like wearing a seat belt or reading a book. Thus, the Gallup question on seat belt usage—“Thinking about the last time you got into a car, did you use a seat belt?”—is superior to the question “Do you ever wear seat belts?” Such wording works only for fairly common behavior. For less common behavior, an interviewer can obtain the same effect by asking about the behavior over a relatively short time period. Thus, instead of asking “Do you ever attend concerts or plays?” the interviewer would ask “Did you attend a concert or play in the past month?” Make the Questions Less Threatening As amusingly illustrated by Barton at the beginning of this chapter, researchers have often attempted to load questions in order to make
Slide 130: 110 ASKING QUESTIONS them less threatening. There have been, however, few controlled experiments to test the effectiveness of these loading procedures, so we are unable to confidently attest to their effectiveness. Nevertheless, we discuss them here since they have some intuitive appeal. For undesirable behavior, the following loading techniques have been used: 1. Use the “everybody does it” approach. The introduction to the question indicates that the behavior is very common, so as to reduce the threat of reporting it. For example, “Even the calmest parents get angry at their children some of the time. Did your child(ren) do anything in the past seven days, since (date), to make you, yourself, angry?” Another version is given in the introduction to Question 2 in Figure 3.8: “The most popular alcoholic beverage in the country is beer.” Of course, the “everybody does it” statement must appear to be reasonable to the respondent. If not, such as in the Barton example—“As you know, many people have been killing their wives these days”—the statement will be ineffective and may actually backfire and increase threat. 2. Assume the behavior, and ask about frequencies or other details. It is usually undesirable to assume that a person is doing something, since a question making that assumption leads to overreporting of behavior. For behavior that is underreported, however, this may be what is needed. For example, a closed question, “How many cigarettes do you smoke each day?” with “None” as a category at the top and with answers ranging from one to forty or more, may reduce the threat of reporting smoking. For financial questions, assuming the presence of assets and asking about details improves reporting. Thus, instead of asking “Do you or members of this household have any savings accounts?” the question is phrased as follows: “Turning to savings accounts—that is, accounts in banks, savings and loan associations, and credit unions—are there separate accounts for different family members or do you have different accounts in various places under the same
Slide 131: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 111 names, or what? Since I have several questions on each account, let’s take each one in turn. First, in whose name is this account? Where is it?” (Ferber, 1966, p. 331). Note that this question also provides memory cues to the respondent. 3. Use an authority to justify behavior. Respondents may react more favorably to a statement if it is attributed to someone they like or respect. An example might be the following introduction to a question about drinking liquor: “Many doctors now believe that moderate drinking of liquor helps to reduce the likelihood of heart attacks or strokes. Have you drunk any liquor in the past year?” It is probably better to use group designations such as doctors or scientists or researchers and not the names of particular persons, since some respondents will not know of the person or may not consider the person an expert. Note that these suggestions to load the question toward reporting socially undesirable behavior would have the undesirable effect of increasing overreporting if the behavior were either socially desirable or nonthreatening. Similarly, the following suggestion for reducing overreporting of socially desirable behavior should not be used with socially undesirable topics. 4. Provide reasons why not. If respondents are given good reasons for not doing socially desirable things such as voting or wearing seat belts, they should be less likely to overreport such behavior. These reasons may be in the form of questions or statements. Thus, on a seat belt usage question the introduction might be “Many drivers report that wearing seat belts is uncomfortable and makes it difficult to reach switches, such as lights and windshield wipers. Thinking about the last time you got into a car, did you use a seat belt?” Another way to ask the question is as follows: “Do you ever find wearing a seat belt uncomfortable? Do you ever have trouble reaching switches such as lights and windshield wipers when wearing a seat belt? Thinking about the last time you got into a car, did you use a seat belt?”
Slide 132: 112 ASKING QUESTIONS Determining the Perceived Threat of Questions It is often very useful to determine at the end of an interview which questions were considered threatening or hard to understand. An example of a series of questions we have used for this purpose is given in Figure 3.10. The most useful of these is Question 4, which asks respondents to indicate whether they thought the questions “would make most people very uneasy, moderately uneasy, slightly uneasy, or not at all uneasy.” Note that this is a projective question about most people and is less threatening than the direct questions asking respondents to report about their own uneasiness. Such questions can be used not only to determine general levels of threat but also as an indicator of respondent veracity. Respondents who report that the question would make most people uneasy are more likely to underreport than are other respondents. Use Additional Sources to Validate Accuracy Although validation from outside sources is always valuable in surveys of behavior, it is particularly important to validate the level of threat associated with a given behavior. As we have seen in this chapter, overreporting and underreporting can be dealt with in various ways, but there is still not enough research to predict in specific cases how big an effect these procedures will have. Moreover, some behaviors such as sexual activity, by their very nature, are private, and no outside validation is possible. Where it is possible, however, validation provides a procedure for evaluating results obtained from alternative methods and ultimately leads to better questionnaires. Validation at an individual level is most powerful but also most difficult. It is possible, for example, to compare individual reports of doctor and hospital visits to records of medical care providers or insurers, but that requires permission from the individual and cooperation from the provider. It must be remembered that record information is also incomplete and variable in quality. Such methods as chemical analysis of hair, saliva, and urine samples have been devel-
Slide 133: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 113 Figure 3.10. Post-Interview Evaluation of Threat. 1. Now that we are almost through this interview, I would like your feelings about it. Overall, how enjoyable was the interview? 2. Which questions, if any, were unclear or too hard to understand? 3. Which of the questions, if any, were too personal? 4. (Hand R. Card W.) Questions sometimes have different kinds of effects on people. We’d like your opinions about some of the questions in this interview. As I mention groups of questions, please tell me whether you think those questions would make most people very uneasy, moderately uneasy, slightly uneasy, or not at all uneasy. How about the questions on: a. Leisure time and general leisure activities? b. Sports activities? c. Happiness and well-being? d. Gambling with friends? e. Social activities? f. Drinking beer, wine, or liquor? g. Getting drunk? h. Using marijuana or hashish? i. Using stimulants or depressants? j. Petting or kissing? k. Intercourse? l. Masturbation? m. Occupation? n. Education? o. Income? p. How about the use of the tape recorder? Source: National Opinion Research Center, 1974. oped to validate reported use of certain drugs. Obviously chemical analysis requires cooperation from respondents, but, perhaps surprisingly, a majority of respondents are willing to provide such samples. Public behavior such as registering to vote and voting can be checked against voting records, and it is sometimes possible to match expenditure data against bills or company records. Validation at an aggregate level is easier if appropriate outside measures can be located. At the very least, it is worth searching
Slide 134: 114 ASKING QUESTIONS carefully for such measures before concluding that no validation is possible. If the behavior involves a product or service, you can compare consumer reports with those of manufacturers, retailers, or suppliers of the service, as with purchases of beer, wine, and liquor. For socially desirable behavior such as giving to charity, you can compare the amounts reported with the total amounts received; reported play and concert attendance can be compared with figures on total tickets sold. But be careful to avoid comparing apples and oranges. In many cases there will be a nonhousehold component in the validation data. Thus, business firms also contribute to charity and purchase goods from retailers. On the other hand, validation data may be useful, even if the comparisons are not perfect. Summary Threatening behavior questions are intrinsically more difficult to ask than are nonthreatening questions. As the questions become very threatening, substantial response biases should be expected, regardless of the survey techniques or question wordings used. For less threatening questions, carefully designed question formats and wording can substantially improve response accuracy. The procedures suggested in this chapter for obtaining more accurate reports of threatening topics include: (1) using selfadministered methods to increase perceptions of confidentiality, (2) using card sorting and randomized response, (3) using openended questions, (4) using long questions with familiar words, (5) using informants, (6) using diaries and panels, (7) embedding the question, (8) choosing the appropriate time frame, (9) making the questions less threatening. For socially desirable behavior, ask respondents about their most recent behavior rather than about their usual behavior. For socially undesirable behavior, ask respondents whether they have ever engaged in a particular behavior before asking about their current behavior.
Slide 135: ASKING THREATENING QUESTIONS ABOUT BEHAVIOR 115 Additional questions are useful at the end of the interview to determine the respondent’s level of perceived threat. Validation, at either an individual or aggregate level, is helpful. Additional Reading Much of the work in this chapter is based on research reported in Improving Interview Method and Questionnaire Design (Bradburn, Sudman, and Associates, 1979; see especially chaps. 1, 2, 5, 9, and 11). Research on the effects of anonymity on responses to threatening drug questions is presented in Turner, Lessler, and Gfroerer (1992). Additional information on the use of audio-CASIC methods can be obtained from the Research Triangle Institute or the National Institute on Drug Abuse. A description of the procedures for validating drug reports is given in Mieczkowski (1999). Wentland and Smith (1993, chap. 4) summarize a broad range of studies on sensitive topics, including alcohol-related behavior, deviant behavior, and sexual behavior. Earlier research on anonymity has been done by Ash and Abramson (1952), Colombotos (1969), Fischer (1946), Fuller (1974), Hochstim (1967), and King (1970). For studies of randomized response see Greenberg and others (1969); Horvitz, Shaw, and Simmons (1967); and Reinmuth and Geurts (1975). The initial paper on this topic was by Warner (1965). Consult also the journals referred to in Chapter Two for other examples of treatment of sensitive topics. In addition, in nearly any given inquiry, specific questioning methods are continuously being investigated. Advances in these areas may be specific to health reporting, alcohol or drug use, financial behaviors, and risk-related activities.
Slide 137: Chapter Four Asking Questions About Attitudes and Behavioral Intentions Setting out rules for formulating questions about attitudes is more difficult than for behavioral questions because questions about attitudes have no “true” answer. By this we mean that attitudes are subjective states that cannot, even in principle, be observed externally. Attitudes exist only in a person’s mind. They can be consistent or inconsistent, clear or unclear, but they cannot be said to be true or false. Thus, in studying the effects of different wordings and different contexts on answers to attitude questions, we have no external standard with which to validate different forms of questions. We must rely on observing how answers may be affected by different factors. Researchers must decide for themselves which form of question is best for their purpose. In this chapter we introduce the principal factors that challenge question writers, and we suggest reasonable solutions. The best advice we can offer those starting out is to borrow (with credit) questions that have already been used successfully. By borrowing questions, you can spare yourself much agony over the formulation of the questions and extensive pretesting. If the questions have been used frequently before, most of the bugs will have been ironed out of them. Also, if the questions have been used on population samples similar to the one in which you are interested, you get the advantage of comparative data. Replication is greatly encouraged, but make sure that the attitude question you borrow is about the attitude you want to study and not about something different. 117
Slide 138: 118 ASKING QUESTIONS Checklist of Major Points 1. Make sure the attitudes you are measuring are clearly specified. 2. Decide on the critical aspects of the attitude to be measured, such as cognitive, evaluative, and behavioral components. Do not assume that these components must be consistent. 3. Measure the strength of the attitude by building a strength dimension into the question itself. Either ask separate questions about attitude strength or ask a series of independent questions, each of which reflects the general attitude. 4. Measure behavioral intentions either directly or by asking about the likelihood a respondent will engage in a behavior. For infrequent behaviors, likelihood measures are best; for frequent behaviors, direct measures are better. 5. Avoid double-barreled and one-and-a-half-barreled questions that introduce multiple concepts and do not have a single answer. 6. Whenever possible, separate the issues from the individuals or from the sources connected with the issues. 7. Consider using separate unipolar items if bipolar items might miss independent dimensions. 8. Recognize that the presence or absence of an explicitly stated alternative can have dramatic effects on response. 9. Specify alternatives to help standardize the question. 10. Pretest new attitude questions to determine how respondents are interpreting them. Using split ballots in pretests is highly desirable. 11. Ask the general question first if general and specific attitude questions are related. 12. Ask the least popular item last when asking questions of differing degrees of popularity involving the same underlying value.
Slide 139: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 13. Ask exactly the same questions in all time periods, if at all possible, when attempting to measure changes in attitude over time. 119 Identifying the Object of the Attitude Attitudes do not exist in the abstract. They are about or toward something. That something is often called the attitude object. The object of attitudes can be practically anything. It can range from being quite specific (such as the President or cornflakes) to quite abstract and general (such as civil liberties or the right to privacy). As with all questions, the cardinal rule is to ask what you want to know, not something else. With attitudes, however, it is more difficult to know what you want to know because the attitude object is often ambiguous or illdefined. The context in which questions are asked has a greater impact on attitude measurement than on behavior questions because the meaning of the questions may be strongly influenced by the context in which they appear. Even preserving the exact wording of an individual question may not be enough to guarantee that two questionnaires are comparable; other questions in the questionnaires may alter the perceived meaning of the questions. The first step in formulating attitude questions is to make sure you know and clearly specify the attitude object. In other words, be clear about what your research question is and what you are trying to find out. In many instances, that requires considerable thought and articulation. For example, consider the following question: “Do you think the government is spending too much, about enough, or too little on homeland defense?” What is the attitude object to which this question refers? One might say at first glance that it refers to government policy toward homeland defense, but which government—federal government, state government, or local government? What is meant by “homeland defense”? Does it include only defense against attack by terrorists?
Slide 140: 120 ASKING QUESTIONS Does it include civil disturbances or rebellions? Does it include defense against natural disasters such as earthquakes, tornadoes, or against electrical grid failures or nuclear power plant explosions? Since many questions contain such ambiguities, extensive pretesting is necessary if you are to develop good standardized questions that eliminate misleading, noisy ambiguities. Unfortunately, because of budget limitations and a belief that question wording is a simple matter that does not require great skill or experience, many researchers do not devote the needed time and effort to pretesting questions. For the pretest phase, Belson (1968) has suggested a technique now known as cognitive interviewing, whereby respondents are asked to restate, in their own words, what they think the meaning of a question to be. This technique is analogous to back translating when questions are translated into another language. Another technique of cognitive interviewing is to ask respondents to verbalize the thought processes they go through when they are answering questions. After using cognitive interviewing, Belson pessimistically concluded that even with well-developed, simplified questionnaires, many respondents do not understand a question in the way it is intended by the researcher. Lack of clarity is particularly common among attitude objects that are frequently discussed in the media, such as “welfare,” “big business,” “civil rights,” and “profits.” For example, in one study discussed by Payne (1951), more than one-third of the population did not know what “profits” meant. Of the remainder, a substantial number had an idea of profits that was quite different from that used by companies who reported them. Fee (1979) investigated the meanings of some common political terms used in public opinion studies. Adopting a variant of Belson’s method, she asked respondents to elaborate on their understanding of particular terms, such as “energy crisis.” She found that at least nine different meanings were attached to the term “energy crisis.” Similarly, the term “big government” elicited four distinct connotations or images: (1) “welfare,” “socialism,” and “overspending”; (2) “big business” and “gov-
Slide 141: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 121 ernment for the wealthy”; (3) “federal control” and “diminished states’ rights”; and (4) “bureaucracy” and “a lack of democratic process.” The images tended to be held by people with differing political orientations or levels of education and were related to different attitudes. Without knowing which of the images respondents held, a researcher might not be able to interpret responses to questions about “big government.” There is no reason to believe that the situation has changed dramatically since then. In short, ambiguity pervades questionnaires. Pretesting and experiments with question wording can resolve some of the ambiguity with regard to respondents’ understanding of questions; but they can do so only if you have a clear notion of what you are trying to find out. If you do not know what you want to know, respondents are unable to help. The Three Components of Attitudes The terms opinion and attitude are not clearly differentiated from one another. In general, opinion is most often used to refer to views about a particular object such as a person or a policy, and attitude is more often used to refer to a bundle of opinions that are more or less coherent and are about some complex object. One might have an opinion about a particular proposal to change a Medicare provision and a more general attitude about Medicare. Opinions are more often measured by single questions; attitudes tend to be measured by a set of questions that are combined on the basis of some measurement model. Attitudes are thought of as having three components: cognitive, evaluative, and behavioral components. The cognitive component consists of a set of beliefs about the attitude object (such as “How healthy is pizza on the following dimensions?”). The evaluative component consists of evaluation of the object; for example, do respondents think it is good or bad or do they like it or not (“Do you like pizza as a food?”). The behavioral component is related to respondents’ attitudes in relation to their actions (“How many
Slide 142: 122 ASKING QUESTIONS times will you eat pizza in the next month?”). In practice, attitude questions are mostly about beliefs and evaluations. Questions that try to measure the action component are discussed later in this chapter. Different Attitude Components Require Different Questions It is generally believed and empirically supported that there is a strain toward consistency among these attitudinal components. People are less likely to believe something derogatory about something they like and are in favor of, and they do not usually act in support of things they disapprove of. The belief that these three components are consistent is sometimes so strong as to lead researchers to neglect assessing the components independently. They assume they can infer other components of the attitude by measuring only one component. For example, respondents who believe a particular product has positive attributes will be favorably disposed to the product and will buy it. Similarly, someone who votes for a particular candidate knows something about the candidate and generally has a favorable view of that candidate. Unfortunately, attitudes are often much more complex and differentiated. Even though components of the attitude correlate in general ways, the components are still different. It is particularly difficult to make inferences about action from simple measurements of the cognitive and evaluative components of the attitude because factors other than the attitude affect action. Even when you are measuring a single component, such as the evaluative component, using different evaluative words may produce different results. Similar (if not synonymous) terms that indicate a positive orientation toward an attitude object may have somewhat different connotations and yield different responses. For example, the terms “approve and disapprove” and “like and dislike” are frequently used in attitude questions, but little attention is paid to possible differences in implication between them. An empirical
Slide 143: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 123 test of the similarities of these terms was obtained in a context of questions about year-round daylight savings time (Murray and others, 1974). The following two questions were asked of the same respondents in a national probability sample in March and April of 1974: As you know, we recently switched from standard time to [year-round] daylight savings time. That means that it now gets light an hour later in the morning than before we switched over. It also means that it now gets dark an hour later in the evening than before we switched over. How do you feel about being on [year-round] daylight savings time now? Would you say you like it very much, like it somewhat, dislike it somewhat, or dislike it very much? As you know, the United States Congress put our country back on daylight savings time this winter as part of a two-year experiment to try to save energy. Some people think that we should continue to have daylight savings time all year round— that is, not turn the clocks back at the end of next October. Would you approve or disapprove of remaining on daylight savings time all year round next year, or don’t you care one way or the other? Although a cross-tabulation of the responses indicated a positive correlation between the two items, 14 percent of those who liked year-round daylight savings time “very much” “disapproved” of it, and 10 percent of those who disliked it “very much” “approved” of it. The correspondence between the two evaluations was highest for those who felt strongly about the issue. The correspondence between the two evaluations was considerably less for those with more moderate likes or dislikes. These findings support the belief that strongly held attitudes are generally more resistant to effects of question wording than are weakly held attitudes.
Slide 144: 124 ASKING QUESTIONS Assessing the Strength of Attitudes Strength is a concept that can be applied to each of the three components of attitudes. Evaluations may be strongly or weakly held, beliefs may be certain or uncertain, and actions may be definitely committed to or only vaguely contemplated. Three general strategies for measuring attitude strength are as follows: (1) build a strength dimension into the question itself by measuring evaluations and strength at the same time; (2) use a separate question to assess the strength; (3) assess strength by asking a series of independent questions, each one reflecting the same general underlying attitude. In this third case, the measure of attitudinal strength is determined by the total number of items a person agrees with. This third method of asking multiple items can be applied to each of the components, although in practice attitude strength is more usually assessed in the general or overall evaluative dimension. Perhaps the most frequent method of measuring intensity of attitudes is to build an intensity scale into the response categories. People’s responses indicate not only the direction of their evaluation but also their intensity or certainty. A common method is to ask respondents a question that measures both the direction and the intensity of an evaluation, as in the first question on daylight savings time quoted in the previous section. This question could have been asked as two separate questions. For example, “Do you like or dislike [year-round] daylight savings time?” and “Do you like or dislike it [year-round daylight savings time] somewhat or very much?” In this case, the simplicity of the like-dislike dimension and the simplicity of the two intensity modifiers suggested that they could be combined into a single question that respondents could easily comprehend. Note that respondents who said they did not care either way were not urged to say which direction they were leaning in. In this case, however, respondents were discouraged from indicating indifference, since a “Don’t care” response category was not included. (This point will be discussed more fully in the next chapter.)
Slide 145: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 125 Using separate questions to evaluate attitude strength is illustrated in another daylight savings time survey. Here the researchers believed that attitude strength was not evenly divided between those who preferred daylight savings time and those who did not prefer it. Therefore, researchers decided to measure separately the general orientation pros and cons and the strength of those feelings. The form of the NORC question is given in Figure 4.1. In this case it was found that those who did not approve of daylight savings time in fact felt more strongly about it than those who did approve of it. Again, those who said they had no preference or did not know were not prodded to see whether they might lean one way or the other. Aggregate Measures of Attitude Another strategy for measuring the strength of attitudes is to combine answers to several separate questions, each one of which is thought to be a measure of the attitude. This method is most often Figure 4.1. Questions on Daylight Savings Time. As you know, the time that we set our clocks to can be changed if we wish. For example, in most parts of the country, we set our clocks ahead one hour in the summer, so that it gets dark at around 9 o’clock instead of 8 o’clock. This is known as daylight savings time. Some people think that we should go onto daylight savings time all year around, that is, turning the clocks ahead one hour and leaving them there. Would you approve or disapprove of going onto daylight savings time all year round, or don’t you care one way or the other? Approve (Ask a.) Don’t care Disapprove (Ask a.) Don’t know a. How strongly do you feel about it? Do you (dis)approve very strongly, pretty strongly, or not too strongly? Very strongly Pretty strongly Not too strongly Source: National Opinion Research Center, 1977.
Slide 146: 126 ASKING QUESTIONS employed to measure general attitudes about abstract objects such as “liberalism” or “freedom of speech.” The general attitude is thought of as a single dimension, usually running from pro to con, or from low to high, or as being anchored at two ends with conflicting orientations. The general attitude is conceptualized as giving rise to many specific opinions about more specific cases. This form of measuring attitude strength often rests on an implicit or explicit mathematical measurement model relating the responses to particular questions to positions on an attitude scale. Public opinion polls typically use single questions to measure either beliefs or more often evaluative aspects of attitudes about objects. An example of this is the classic Gallup presidential rating question, “Do you approve or disapprove of the way [name of president] is handling his job as president?” In contrast to the single question approach, scientific and many commercial surveys often use several questions to measure the different aspects of an attitude, and they then combine the questions in a scale that allows for greater differentiation. If these multi-item scales are carefully constructed, they can produce a more reliable measure of attitudes because they filter out a lot of random “noise” that contributes to measurement error. Likert Scales and Guttman Scales A number of scaling techniques have been developed based on somewhat different underlying models of attitude measurement. The most popular is the Likert scale, named after Rensis Likert, a pioneer in the field of attitude measurement. The fundamental idea behind Likert scales is that an attitude can be thought of as a set of propositions about beliefs, evaluations, and actions held by individuals. If you ask respondents to agree or disagree with a sample of propositions about the attitude object, you can combine the answers to get a better measure of the attitude. A number of different statistical techniques are used to determine whether a particular set of questions actually forms a scale (see Additional Reading),
Slide 147: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 127 but they all rely on the idea that a good scale has a high intercorrelation of the items. Another commonly used scale type is the Guttman scale, named after Louis Guttman, another pioneer in attitude measurement. Guttman’s approach was to devise questions that measured increasing agreement (or disagreement) with attributes of the attitude object. In contrast to the Likert model where the total number of items agreed to without regard to their order is the measure of the attitude, items in a Guttman scale are ordered such that some items should be agreed to only by those who are low on the attitude and others should be agreed to only by those who are high on the attitude scale. An example of the difference in approach applied to attitudes about free speech is presented in abbreviated form in Figure 4.2. The first format is that of the Likert scale. Several items are stated in propositional form and respondents are asked to indicate the degree of their agreement or disagreement with the statement. The scale score is then determined by giving numerical values to the response categories and adding up the values given to respondents’ responses. The second format in Figure 4.2 is that of a Guttman scale. Here the items are presented in an order of increasing commitment (or opposition) to free speech. Various methods for combining responses can be used. The simplest is to count the number of “yes” and “no” answers as appropriate. Some decision would have to be made about the “don’t knows.” One alternative is to leave them out or to treat them as lying somewhere between the “yes” and “no” responses. More complex treatments include giving more weight to the less frequent responses or weighting the less frequent responses according to some a priori view of the relationship between the content of the hypothetical speech and the strength of a belief in freedom of expression. Again we stress that you must make such decisions on the basis of your research question and your conception of the measurement model. Guttman scales are somewhat harder to develop and thus are not so commonly used as Likert scales.
Slide 148: 128 ASKING QUESTIONS Figure 4.2. Two Formats for Measuring Attitude Toward Free Speech (Abbreviated Questions). LIKERT FORMAT Please tell me how strongly you agree or disagree with the following statements. Do you agree strongly, agree somewhat, disagree somewhat, or disagree strongly that . . . 1. Communists should be allowed to teach in school. 2. People should be allowed to make speeches supporting the view that whites are genetically superior to other races. 3. Books advocating the violent overthrow of the government should be banned from public libraries. GUTTMAN FORMAT Consider a person who believes that whites are genetically superior to all other races. If such a person wanted to make a speech in your community claiming that whites are genetically superior to blacks, should he be allowed to speak or not? Yes, allowed Not allowed Don’t know Should such a person be allowed to teach in a college or university or not? Yes allowed Not allowed Don’t know If some people in your community suggested that a book he wrote which said that whites were genetically superior should be taken out of your public library, would you favor removing this book or not? Favor Not favor Don’t know
Slide 149: ASKING QUESTIONS ABOUT ATTITUDES AND BEHAVIORS 129 It is beyond the scope of this book to deal in detail with these models, but it is important to know that explicit measurement models often provide criteria to help measure attitude. Working back and forth between pretesting questions and testing responses against an explicit measurement model can greatly aid in the development of valid attitude scales. Consult one of the references listed at the end of the chapter for more information about measurement models. Many researchers create scales without properly testing whether the scales meet statistical criteria for a good scale. Researchers are well advised to explicitly articulate the measurement model they are using and to get appropriate statistical consulting advice when they are in doubt about how to refine and test their scaling procedures. Asking Behavioral Intention Questions There is often an assumed relationship between attitudes and behaviors. That is, people are asked attitude questions about products because their attitudes might reflect what they buy, and people are asked their opinions about political candidates because their answers might reflect how they will vote. When behavior is impossible to measure, such as when it relates to an unreleased new product or an upcoming presidential campaign, we often use attitude measures as a surrogate for behavior. A basic and frequently useful conceptualization of the attitudebehavior connection uses the intermediate step of behavioral intentions. This view basically contends that attitudes are related (to some extent) to behavioral intentions, and these intentions are, in turn, related to actual behavior (see Figure 4.3). Certainly there are a lot of other factors in the equation. In the final analysis, however, a good predictor of behavior is the consistency of behavioral intentions and attitudes toward the object. For instance, respondents’ attitudes toward a soft drink might be related to their intentions to buy that soft drink within the next week, and these intentions might be related to actual purchase.
Slide 150: 130 ASKING QUESTIONS Figure 4.3. Behavioral Intentions Can Link Attitudes and Behaviors. Behavioral Intentions • Likelihood • Frequency Attitude Behavior Similarly, one’s attitude toward exercising and eating less might be related to the intention to “lose ten pounds before my high school reunion,” and this intention is likely to be related to actual weight loss. Although a number of other factors make these relationships imperfect, it is often believed there is a general mapping from one’s attitude to behavioral intentions. One important reason for measuring behavioral intentions (instead of just attitudes) is that behavioral intentions can help differentiate between people who choose an endpoint on a scale. For instance, if two people really like a product and rate it as a 9 on a 9-point scale, there is no way to determine whether one plans to consume the product three times or thirty times within the next year. Behavioral intention questions can be used to estimate how likely one is to perform a behavior (buy a car in the next year, or donate money to a presidential candidate’s campaign). They can also be used to estimate how frequently one is likely to perform a behavior (exercise in the next year, call long distance in the next month), and so forth. Both are of interest in a number of cases. In this section we will discuss how and when it is important to ask one type of question versus the other. Asking Likelihood Questions Regarding Incidence “Do you plan on buying a new car in the next year?” This is an incidence question that does not focus on how many cars that person will purchase in the next year, but instead focuses on how likely the respondent is to buy a car. The two most common types of behav-