sample="quota" bates="2057063728" isource="pm" decade="1990" class="ui" date="19951223" EVALUATION OF METHODOLOGICAL and CONTENT ISSUES FOUND IN FOCUS GROUP TRANSCRIPTS AND THE SUMMARY PROVIDED IN THE FEDERAL REGISTER of DECEMBER 1 EDWARD V. MORSE, PhD HOWARD J. OSOFSKY, MD, PhD AND PATRICIA M. SIMON, MSW, PhD December 23, 1995 INTRODUCTION Focus groups, as a research technique within the scientific tradition, serve a limited but valuable function of assisting the behavioral scientist in the generation of hypothesis and the development of insights into patterns of human behavior from which grounded theory can be constructed. Save for their use by those interested in product marketing, the focus group technique is normally used in conjunction with research methodologies that generate more quantitative data. The very unstructured nature of focus group data generation, collection, and analysis makes it a technique primarily for the exploratory phase of a study, wherein the researcher is interested in identifying the qualitative relationships between an array of attributes. Conversely, however, it is this same unstructured quality that makes data generated by means of focus groups extremely susceptible to invalidation of content when methodological rigor is maintained. Among the methodological considerations that are of particular concern are: adequate sample size, representativeness of the sample, instructions given focus group participants, and the nondirective interviewing techniques used by the focus group moderators. This report addresses two sets of concerns, one regarding issues of content and the other issues of methodology, arising from the focus group testing conducted for the FDA to help it evaluate the content and various formats for possible additional warning statements the agency may prescribe under proposed section 897.32(c). As noted above, under the best of circumstances, focus groups provide only qualitative data. Therefore, there is good reason to seriously question whether data coming from a very limited set of 19 focus groups with a total combined n of 149, as is the case here, should be used as any basis for determining the effectiveness of the proposed additional warning (60 Fed. Rg. 41314 at 41338). The focus group data that are intended to be used by the FDA to evaluate the potential effectiveness of a public policy are seriously compromised. In some instances the methodological concern is serious enough to invalidate use of all of the data in a single focus group. In other instances, the methodological concern clearly suggests that it would be scientifically prudent of the FDA to eliminate, from any further use, large sections of data from a significant number of the focus groups. In one instance the focus group data appear to reveal a fatal error endemic to all 19 focus groups, which if substantiated, should give rise to the elimination of all the focus group data. Second, in the process of reading the transcripts of the 19 focus groups it became clear, to all three of the independent scientists reading, that a major finding articulated repeatedly in many of the groups was not presented in summary of the focus group findings as reported in the Federal Register December 1, 1995 (60 Fed. Reg. 61670). In the summary of the focus group data contained in the Federal Register, it is clearly recognized that: " Most of the participants indicated that they did not believe that they were influenced (to smoke) by cigarette advertisements" (see Section II-Perceptions of Cigarette Advertising, no.3) and quotations from the focus group transcripts to that effect are presented. Missing from the report in the Federal Register however, is an equally critical observation arising from the data regarding the perceived impact of "anti-smoking" advertising on individuals' behavior. The focus group members clearly articulated, on numerous occasions, that they felt anti-smoking advertisements would not prevent individuals from smoking or stop those who had already chosen to smoke. It is important to keep in mid that the participants repeatedly stated that no matter what color, size, or shape the advertisement took, anti-smoking advertising would not change behavior. The remainder of this evaluation examines specific data from the focus group transcripts by first drawing attention to methodological issues and later by presenting data that reflect a significant unreported finding regarding the inability of anti-smoking advertisements to alter behavior. In each section the concerns presented by the material are addressed along with how each might impact the validity of the data collected. Additional findings from the focus groups that were not reported in the Federal Register will clearly demonstrate the fallacious assumption upon which FDA is proceeding. CONCLUSION: THERE ARE NO METHODOLOGICAL GROUNDS UPON WHICH TO JUSTIFY THE USE OF DATA GATHERED FROM CHILDREN LESS THAN 18 YEARS OF AGE, FROM WHOM ASSENT HAS NOT BEEN ATTAINED. THUS THE DATA FROM EITHER AGE GROUP #1597 MUST BE ELIMINATED OR ALL DATA SHOULD BE RENDERED INVALID. 5. Repeatedly the moderators are not neutral in the questions they ask and on many occasions they directly shape the responses they are asking for by the content of the questions asked. Instances of the interjection of moderator bias are too numerous to document because it is endemic to the transcripts in general. Bias is so consistent as to call into serious question the usefulness of these data for any scientific or policy purpose. Typical examples are: "Moderator Bryant: Okay. All right. The warning labels that we've seen so far are not the only possible warning labels, and you guys have come up with some ideas of your own. I've got here some possible warning labels that people behind the glass there have put together, and what I'd like you to do is take a look, first of all, at the top sheet, we are going to go through each sheet separately, so if you can just leave them in order, if you can pass these down." (p.44, vol. 1587) " Moderator: , you've been quiet, what do you think? Participant: Well, I don't know. Moderator Davis: You don't know because you don't care or don't know because there's just not enough information (inaudible)? Do you have an opinion or -- Participant: I don't really (inaudible) Moderator Davis: It sounds like, maybe you would vote that this would probably not very very effective. Like, you could do it, but, it may not really change very many people's behavior." (p.32, vol. 1596) "Moderator Bryant: Okay. One of the things that nobody brought up as far as why kids smoke was advertising." (p.26, vol. 1598) "Moderator Bryant: So, we told you 8:30, but I'll try and let you out early if everybody cooperates with me. All right." (p. 9, vol. 1600) "Moderator Bryant: Okay. How about, you said like 99 percent of kids would try smoking, or 75 to 99 I heard, how about smoking marijuana, what percent do you think in your age group would have tried marijuana at some point?" (p.15, vol 1600) "Moderator Bryant: Okay. I'd now like to kind of switch gears completely and just talk about advertising and first of all let's just talk about advertising in general, and forget cigarette advertising, but when you think about compelling advertising, what kind of an ad would you say is compelling?" (p. 12, vol. 1588) CONCLUSION: MONITORS SET SUCH A BIASED TONE IN CONDUCTING FOCUS GROUPS AS TO RENDER THE DATA GENERATED USEFUL FOR EXPLORATORY PURPOSES ONLY, AND OF NO VALUE TO SETTING POLICY. Specific Issues 1. The moderator threatens to reduce the pay of the focus group participants if they do not cooperate. It is important to note, in this case, the impact this threat has on the participants. In focus group #1603 page 65-66 the participants begin trying to find out for whom the moderator works and the moderator keeps putting them off. Rather than answer their question so that the group could get back on task, the moderator, from his position of power, directly threatens the group. The impact the threat has is clear as the following excerpt shows: Moderator: All right, if we could, get back to this. I will tell you, you know your pay will go down if you don't cooperate on this. you can-- Participant : I'm done. Participant: You got me. Participant: I think that's the perfect size. Participant: Yeah. Clearly the participants were ready to agree with whatever the moderator wanted just to keep their pay. CONCLUSION: ALL DATA COLLECTED IN THIS FOCUS GROUP AFTER THE THREAT BY THE MONITOR SHOULD BE EXCLUDED FROM ANY PRESENTATION OF THE FINDINGS. IT WOULD BE WHOLLY INAPPROPRIATE FOR THE FDA TO BASE ANY RULING ON DATA GATHERED IN SUCH A MANNER. "I don't think anything is going to help. Everybody is just going to read it, they'll read it and they'll make up their mind that they're going to do it or they won't"(p.59, vol. 1590). "It won't do anything...If they want to do it they want to do it" (p.72, vol. 1590). "I don't think that I mean writing would actually have any affect. I think they'd have to be like shown" (p.52, vol. 1594). "I think no matter what you do to the warning, it's' still like we're say on the sheet here, it doesn't...nobody really cares about the warning, no matter what you do to it, no matter how you dress it up. Nobody really cares about it" (p.53, vol. 1594). "Nobody really trusts the government anyway"(p. 54, vol. 1594). "I don't think a warning is going to do any good, but I just think the peer pressure is more of a factor when you're younger" (p.56, vol. 1594). "Most kids just think they're immortal. They think nothing can hurt them" (p. 56, vol. 1594). "I don't think the ads have anything to do with whether or not kids smoke" (p.39, vol. 1594). "I think people know the facts. They make the decision for themselves" (p.60, vol. 1594). "I think kids are going to do it (smoke) no matter what. Every kid is going to try it or most kids, at least try it. No matter what the ad says, no matter what their parents say" (p.61, vol 1594). "...It had nothing to do with the ads. They don't keep me from it, nor they don't make me do it" (p.80, vol 1593). "I don't think it'll (additional anti-smoking message) help either (first time or current smokers)" (p.43, vol. 1597). "That wouldn't matter. It (the ad) wouldn't have any effect on me or any of my friends"(p.33, vol. 1591). When asked how many people this ad would keep from picking up a cigarette, the following responses were recorded: "Not many" (p.88, vol. 1597). "I don't think anybody" (p.88, vol 1599). "I think it would take a friend or something like that to convince you, (not to smoke) not a sign" (p. 73, vol. 1599). "I think that if you are going to try it, you are not going to pay any attention to what anything says about it" (p. 76, vol. 1599). "If you really want to do it, it doesn't matter what they say" (p.73, vol. 1599). When examining the transcripts of the focus group participants' responses to the presentation of various anti-smoking slogans and visuals, it is important to separate out the participants responses to the aesthetics of the advertisements in terms of color, size, and graphics from their responses to the ability of these ads to alter behavior. In the focus group summary [Docket No. 95N-0253] these two separate meanings in efficacy are not separately addressed, in fact, they are confused. For example, in Section IV -- "Perceptions of New Brief Statements" the contractor states "A 'positive' response was defined as one where participants indicated that teens their age would read the message, consider the implications of the message, and perhaps change behavior because of it." This statement combines two meanings of efficacy and misleads the reader into thinking that the focus group respondents indicated that the anti-smoking ads would attract attention as well as change behavior. A careful reading of the transcripts indicates that the group participants spent the majority of their time discussing the aesthetics of the ads in terms of their ability to draw a viewer's attention. In fact, as noted above when group participants were asked if these anti-smoking ads would change behavior most said, no. CONCLUSIONS The methodological issues presented above are serious flaws which critically compromise the validity of any finding reported in the focus group summary contained in the Federal Register. Additionally, the failure to report a major finding clearly present in the focus group data further indicates that commonly accepted standards of scientific objectivity were not followed. Thus, it must be concluded that if the FDA is serious in its intent to evaluate the impact of anti-smoking advertising by means of data collected from focus groups, the present data should be discarded and the focus groups conducted again. Submitted by: Edward V. Morse, PhD Howard J. Osofsky, MD, PhD Patricia M. Simon, MSW, PhD 12-23-95 Date