October 21, 2008
Proposition 8: A Closer Look At Those Poll Numbers
Ever since California county clerks began issuing marriage licenses to same-sex couples last June, Proposition 8 — the proposed constitutional amendment to eliminate marriage equality — has appeared likely to lose at the ballot box. Throughout the summer, statewide surveys from the Field Poll and the Public Policy Institute of California (PPIC) consistently found that the measure lacked majority support. In fact, it was opposed by more than 50% of likely voters.
But earlier this month, a new poll of Californians’ voting intentions on Proposition 8 was released, sponsored by several CBS local affiliates and conducted by SurveyUSA. Here’s a section of the news report on the poll findings:
According to the poll, likely California voters overall now favor passage of Proposition 8 by a five-point margin, 47 percent to 42 percent. Ironically, a CBS 5 poll eleven days prior found a five-point margin in favor of the measure’s opponents.
The only demographic group to significantly change their views during this period were younger voters — considered the hardest to poll and the most unpredictable voters — who now support the measure after previously opposing it.
Last week, the same survey organization released new data showing that the ballot race is a statistical dead heat, with 48% supporting Prop. 8 and 45% opposing it (the margin of error is +/- 4 points).
In combination, these polls indicated a surprising shift in California opinion, and they received a lot of media attention. The No On 8 campaign publicized them to warn Californians that Proposition 8 has a serious chance of passage and to galvanize marriage equality supporters to donate to the campaign. Many couples moved up their wedding date in order to have their marriage officially recorded before the election, banking on Attorney General Brown’s opinion that Proposition 8, if passed, won’t apply retroactively.
A Closer Look At Those Data
I certainly agree that the race will probably tighten as November 4 approaches. A September Field poll revealed that voters who haven’t previously thought much about Proposition 8 can be swayed by the measure’s wording on the ballot. And the multi-million dollar media campaign that is now being waged is likely to have already influenced some voters. Thus, in the next two weeks, undecided voters will make their choice, and some “soft” supporters and opponents of the measure will change their minds.
So marriage equality supporters can’t afford to be complacent. Nevertheless, I’d like to register my skepticism about the recent polling data.
The polls showing Prop. 8 winning or tied were conducted by SurveyUSA, a New York firm that specializes in automated surveys. These are surveys that don’t use a human interviewer. Instead, when someone answers the phone, they hear a prerecorded voice (perhaps a local newscaster) asking them to use the telephone keypad to register their answers to a few questions.
According to its website, SurveyUSA:
is the first research company to appreciate that opinion research can be made more affordable, more consistent and in some ways more accurate by eliminating the single largest cost of conducting research, and a possible source of bias: the human interviewer.
The company claims that its polls are superior to others because they eliminate the variability in quality that results from having human interviewers.
A point not mentioned on their website is that many people have a tendency to hang up when they answer the phone and hear a recorded voice. Another problem is that, because no human interviewers confirm a respondent’s eligibility, anyone who answers the telephone — a child or teenager, a babysitter, an out of town guest — can complete the survey. These factors might affect the accuracy of SurveyUSA data.
On the other hand, survey participants are sometimes more willing to reveal potentially sensitive or embarrassing information about themselves to a computer than to a human interviewer. A 2006 study, for example, found that adults were more likely to acknowledge having had sex with a person of their same sex when the interview was conducted via computer rather than with a live interviewer. But it’s not clear that a significant number of Californians are reluctant to say whether they support or oppose Prop. 8. And, unlike SurveyUSA polls, the 2006 survey was first introduced and explained to respondents by a live interviewer.
Comparing Polls
To repeat a recurring theme of this blog, any poll is merely a snapshot of public opinion at one moment in time. Because opinion can shift and polls vary in their accuracy, it’s important to evaluate a survey’s findings in comparison to other polls that addressed the same topic.
So here are the SurveyUSA findings:
June 9: 44% support, 38% oppose (18% undecided)
September 25: 44% support, 49% oppose (7% undecided)
October 6: 47% support, 42% oppose (10% undecided)
October 17: 48% support, 45% oppose (7% undecided)
For comparison purposes, here are the results from the Field and PPIC polls:
May 28 (Field): 42% support, 51% oppose (6% undecided)
July 18 (Field): 42% support, 51% oppose (7% undecided)
August 27 (PPIC): 40% support, 54% oppose (6% undecided)
September 18 (Field): 38% support, 55% oppose (7% undecided)
September 24 (PPIC): 41% support, 55% oppose (6% undecided)
All of the polls show support for Proposition 8 hovering in the low-to-mid 40s, except the most recent SurveyUSA data, which have it gaining – but still not at 50%.
Where they disagree the most is in their estimates of the proportion of Californians who oppose Prop. 8. The Field and PPIC polls put opponents in the majority. Survey USA has them ranging between 38% and 49%.
Election polls can differ for many reasons, including how questions were worded, when the poll was fielded, and how the survey organization defined what constitutes a “likely voter.” Activists and pundits have assumed that the dramatic differences between the SurveyUSA polls and the PPIC and Field polls reflect a real shift in public opinion. They attribute that shift to the Yes On 8 media campaign — which has included images of San Francisco Mayor Gavin Newsom gleefully proclaiming that marriage equality is here “whether you like it or not,” as well as inaccurate claims that failing to pass Proposition 8 will lead to the harassment of churches and pro-gay indoctrination of children.
Those ads may indeed have affected undecided voters and Californians whose opposition to Prop. 8 wasn’t strong. But my guess is that the big difference between SurveyUSA and the Field and PPIC polls lies in the quality of their samples.
The Field and PPIC polls are conducted over a longer time period. This usually results in better data because more people in the original sample are reached than is possible when a poll is in the field for only one or two days. When Field and PPIC researchers don’t reach a respondent on their first attempt, they phone back at least five times.
With overnight and 2-day polls, by contrast, anyone who isn’t home on the first or second try is simply dropped from the sample. This creates an accuracy problem if those people differ from the poll respondents in a way that is relevant to the survey question — in the case of Prop. 8, for example, if they are younger or more liberal than the people who happened to be home when the phone rang. Pollsters try to account for the missing respondents by mathematically weighting the data, but this always involves guesswork.
A second strength of the Field Poll sample is that it’s drawn from official voter registration lists, and respondents are contacted through the address and phone number they provided when they registered. So the Field Poll sample includes registered voters who rely mainly or exclusively on a cell phone. By contrast, the SurveyUSA and PPIC samples were selected through random-digit dialing (RDD), a method widely used in telephone surveys. Individuals without a land line are almost always excluded from RDD samples.
Are cell phone users different from other voters in a way that is relevant to Prop. 8? Maybe. Data from the Pew Research Center suggest that samples based exclusively on land-line calling probably underestimate support for Barack Obama by about 2 percentage points because Obama has a disproportionate amount of youth support, and many young voters rely exclusively on cell phones.
More to the point, Pew researcher Scott Keeter and his colleagues used data from a national survey that included cell-phone users to compare the opinions of adults under 30 on a range of topics. They found that only 37% of young adults in land-line households supported marriage equality, compared to 51% of those who relied on cell phones.
Keeter also found that Pew statisticians were able to effectively compensate for this discrepancy by applying statistical weights to the data from land-line surveys.
We don’t know if SurveyUSA weighting procedures are comparable.
Relevant to this point, the recent SurveyUSA polls have found that voters under 35 either strongly support Proposition 8 (by a margin of 53% to 39% in the October 6 poll) or are closely divided (44% Yes to 46% No in the October 17 poll). By contrast, most polls in California and elsewhere have found that young adults generally favor marriage equality. In my mind, this raises some doubts about the accuracy of the SurveyUSA data.
On Balance…
My own guess is that the Field Poll, by virtue of its sample drawn from the voter registration rolls, probably describes current opinion among California voters more accurately than the SurveyUSA data. The fact that the PPIC poll data have closely tracked the Field Poll results over the summer further boosts my confidence in both surveys. I suspect that the SurveyUSA data are more or less accurately stating the number of likely voters who support Prop. 8, but are undercounting the number who oppose it.
But this is only an educated guess. We’ll have a better sense of current California opinion on the ballot measure when the next round of PPIC and Field Poll results are released, which is likely to be soon.
Meanwhile, even if the SurveyUSA results have important limitations, they highlight two key facts.
First, many Californians are still making up their minds about Prop. 8 and can be swayed during the next two weeks by conversations with their friends and family, endorsements by public figures whom they respect, and media campaigns. This means it’s important for marriage equality supporters to continue to donate to the No On 8 campaign, to declare their opposition to Prop. 8 in public forums, and to personally explain their views about the measure to their friends and family.
Second, voter turnout will be critical. Regardless of whether most Californians support or oppose Proposition 8 today, what truly matters is who votes — whether by absentee ballot, in early voting, or at their polling place on November 4th.
Thus, equality supporters should make sure that their friends, family, roommates, classmates, coworkers, and neighbors vote.
* * * * *
For more information about surveys and cell phones, see the article by Scott Keeter et al., “What’s Missing from National Landline RDD Surveys?: The Impact of the Growing Cell-Only Population,” in Public Opinion Quarterly, 2007, vol. 71(#5), pp. 772-792.