April 15, 2013


Questions I Hate on Surveys: A Continuing Series

I spend a lot of time on the Internet, as you probably do as well. It is there in the background when I am working, it is in my pocket when I am walking, it is playing music in the kitchen as I prepare dinner, and it is sending me alarms and notices. When I am working, my documents are even being synchronized with my cloud services. So, how am I supposed to answer a questions such as “How much time do you spend using the Internet in an average day?” Usually the question asks for the time in hours and minutes.

Sometime in the last ten years, the Internet evolved from something that could be thought of  as a task (viewing a website) to an integral part of other tasks, which makes the question virtually impossible to answer unless it is meant as a measure of Internet browsing. And, if that is the case, maybe we should just ask that question.

To be fair, the reason for the question is pretty obvious. It is a measure of a number of important dimensions: use of technology, engagement with the broader world, and time spent without in-person social interaction. And while it is possible to measure these with Likert scales they lack the time-based scale that gives Internet use questions their perceived power (e.g. they can be compared to the time spent with family, time spent watching tv, etc.). Of course, the fact is that the scale is not reliable and the question lacks meaning.

It is time to retire the question. How the Internet is used and how important it is to other activities are much more useful questions. The fact is that when a person is using the Internet they are always engaged in some specific activity — when a person is spending time on Facebook it is not the Internet that is the most important element of what they are doing.

“How much time do you spend using the Internet?”  suffers from being imprecise for respondents and lacks a clear conceptual purpose. When you ask about time, make sure that it is something that can be measured in a discrete sense.


March 12, 2013

1 Comment

An Historic Evolution of Public Opinion: A Look Back at Same-Sex Marriage in Canada

Public opinion change is not unheard of but when it comes to fundamental beliefs and values, we expect change to take place slowly, if at all. The evolution of Canadian beliefs about same-sex marriage provide an interesting example of dramatic changes that both presupposed and reacted to court decisions.

 Courts played a key role in Canada but public opinion was not an idle bystander.

This is telling as Americans look to an upcoming Supreme Court decision about same-sex marriage that will no doubt shape the political discourse there.

Court decisions were certainly critical points in the evolution of public opinion on the issue. For timelines of the issue see the CBC website. Notably, courts ruled in the early 90s that discrimination based on sexual orientation was protected, that same-sex couples have the same  benefits and obligations as opposite-sex common law couples (Supreme Court, 1999), and in 2002 the Ontario Superior Court rules that prohibiting same-sex marriage is unconstitutional. Ruling on same-sex marriage in other provinces followed until eventually the Parliament of Canada passed a law defining marriage to include same-sex relationships in 2005.

Throughout the period of the 1990s Court decisions were in line with and reflected a growing tolerance toward homosexuality. Attitudes toward Gays and LesbiansThankfully, the Canadian Election Study (CES) and Environics both tracked the evolution. Starting in 1987 Environics asked as part of its Focus Canada study, Canadian attitudes toward homosexuality. In 1987, 10% indicated that they approved of homosexuality (there were many unsures (30%) this year and in 1996). By 2004, this number reached 47%. Even setting aside the very low year of 1987, the increase from to 47% is remarkable. In a different approach, the Canadian Election Study asked respondents to rate homosexuals (Gays and Lesbians after 1993) on a scale from 0 to 100. The average rating for each survey rose from 45.5 to 73 out of 100 in 2011.The outsider nature of the community has broken down substantially in the last decade, particularly after 2004 (between 2004 and 2011 the average rating has increased much more than it did between 1993 and 2004).

Although history seems like inevitable march toward granting and accepting same-sex marriage, in 1999 a federal vote maintained the traditional definition of opposite sex relationships. While courts and Parliament were extending the same rights, the actual inclusion into the institution of marriage was being denied. Public opinion was divided and offered little in specific guidance.

The chart below shows polling on the question over time from four polling firms. There were many other polls but often the questions were only asked once and this provides a very interesting as it is.

Gay Marriage-Trends

The CES surveys, which cover the longest period, show agreement that Gays and Lesbians should be allowed to marry rises significantly between the 1997 and 2000 elections. Note that in this period, the Supreme Court had ruled that same-sex couples have the same benefits and obligations as opposite-sex ones. The question of marriage was definitely on the agenda. Nevertheless, just 50% supported marriage at this time.

As the issue heated up, more polling was taking place and polls showed relatively little movement in attitudes. Most trend lines were not moving dramatically in either direction but overall most were showing support for same-sex marriage hovering around 50% depending on whether respondents were explicitly given an unsure or neither category (this is why the Ekos trend line is lower). All trend lines tend to be more positive once Parliament acted in a manner consistent with the court rulings.

One reading of the trend is that public opinion was divided but leaning toward accepting same-sex marriage but this misses a key element of public opinion at the time — the overall acceptance of same-sex relationships.

The questions in the figure above show the issue in terms of a binary choice — same-sex marriage or not. The reality is that There were fewer staunch opponents than it may seem. A TNS survey in 2004 found that 39% would support marriage and another 35% would support legal unions for same-sex couples.

Moving forward to recognize same-sex relationships in formal-legal way (possibly including marriage) was something that Canadians were overwhelmingly onside with. Courts said it had to be civil marriage and a healthy majority of Canadians now agree.

Canadian courts played a nontrivial role in bringing about same-sex marriage but they did so largely with public support. If courts were leading they found increasingly receptive audiences for their messages.

We are fortunate that the Canadian Opinion Research Archive is around to help preserve the cultural legacy of the public opinion research that has taken place in the past 3 plus decades. I draw on some of the data here along with searches of individual company websites.

February 15, 2013


Why Senate Reform is Not About Public Opinion Right Now and Not Very Likely

Senators have been having a tough time the last few weeks and the institution is looking a lot less like an institution of sober second thought and more like a frat house. But, even if Harper is now regretting his choices and even though he may want senate reform, the recent events do nothing to help motivate change.

Changing the Senate is fraught with risk as any potential change threatens the balance of political power in this country. 

The arguments for reform certainly have some merit on their face. Consider the following reasons: (1) it is a waste of money given the work that is done by Senators; (2) Senators are not accountable to anyone, even the person who appointed them; (3) the institution does not fulfill its mandate to represent the provinces; and, (4) the Senate does not sufficiently exercise independent oversight of House activity. This laundry list is of my own, but it captures much of the debate.


Faced with arguments such as this and the probable situation that Canadians do not fully see the benefits to a Senate, we would not expect much support within the public for the status quo.

A simple question about its future, in this case abolition or not, does elicit a preference for change. Almost half of Canadians (49%) agree that the Senate should be abolished against only 33% who disagree. Taken from the 2011 Canadian Election Study, this provides directional support for change but it is not overwhelming support. And, of course, abolition is the easiest choice because it does not open up the debate about how to reform or change the institution.

Of course, debate over the future of the Senate is not just being framed in terms of the current Senate or no Senate. Reform is another option and when Environics included this option in polling over the past twenty years it found a plurality is in favour or reform rather than abolition.  In fact, 27% would prefer to leave it as it is in the last wave of the study. Interestingly, there has been virtually no movement in public opinion since 1992. 

ImageThe issue becomes even more complicated when we start probing the nature of the reform that Canadians would support. Our strong commitment to democracy means that 86% would prefer that a reformed Senate be elected directly by the people (Environics, Focus Canada, 2007). No doubt it is this underlying democratic commitment that is generating some of the current appetite for reform. We did not choose these people and they are not representing us.

There is, however, no consensus on the composition of the Senate and this is, perhaps the most difficult reform to address. Originally, the Senate was meant to be a place of regional representation (thus representation by population was eschewed for provincial/ regional representation). In the same Environics surveys, when asked to choose between having the an equal number of senators from each province or having the number for each province based on the size of the population, Canadians are slightly more likely to prefer population based allocation, but 41% would prefer equal number per province.

Reform, other than the easy step of letting the people decide, is then fraught with political division within the public.

The challenge in reforming the Senate, is that public opinion offers, at this time, so little in terms of helping to navigate the critical details about how an elected Senate would function and be composed. These details are likely to involve an elite-led process unless the Supreme Court rules that some aspects (e.g. the change from appointed to elected) of  Senate Reform is wholly within the Federal jurisdiction. Baring this, the requirement for sufficient elite consensus across the country seems lacking.  

The public can get annoyed with the current system and tell pollsters they prefer an elected process for the Senate but they are only one player and their annoyance is unlikely to turn into sufficient priority (as compared with more pressing needs such as the economy or health care) to motivate the political class.

More importantly, as yet, the public is insufficiently engaged to truly tell us how they envision the future of the institution. Such as vision, must reflect a true understanding of the consequences for how democracy and government will work in Canada. Such a vision is the only real way Canadians could drive the process for reform.

Public opinion will no doubt matter in the playing out of the politics of institutional reform but one expects it is more likely to act to quash options (and maybe even compromise) rather than directing how political leaders try to sort reform out.

For those interested in reform, some form of deeper citizen involvement and engagement will be necessary. A citizen assembly seems most promising but even here the issues of substantive Senate reform — beyond abolishment — raise potentially intractable issues of political power.

What do you think? Do you want or hope for Senate reform? 


November 5, 2012


Second Term Presidents: Obama’s Legacy

American’s are finalizing their decision on who will become the next President but it is worth considering what this election means for Obama. Can he achieve a second term and what does it mean if he does not?

Incumbents have an inherent advantage when it comes to re-election and Presidents have a fairly high success rate in achieving re-election for a second term.

Since 1968, there have been 6 opportunities for a President to be elected for two consecutive terms. In four of those, the incumbent was successful. Carter (1980) and George H.W. Bush (1992) were the exceptions. Unpacking why some are successful and others are not is clearly not possible in a statistical sense given all of the possible reasons why Americans might re-elect an incumbent but history has its lessons.

What, then, can other recent cases tell us about the task that Obama has set for himself in getting re-elected. Perhaps it is worth considering the previous cases of success and failure.

Notably, all four successful incumbents improved their share of the popular vote and their share of the votes cast in the Electoral College.  It was a bigger win the second time around for the re-elected incumbents. Each of the cases has a story but some part of that story may be that when faced with a reasonably successful incumbent, the opposition party simply does not put up the strongest possible contender.

At 68% of the Electoral College vote (53% of popular vote), Obama won the 2008 election with a fairly comfortable margin. My sense is that the expectation over the past two years was that Obama would win re-election. Recent polls over the weekend suggest that Obama will not grow or retain his 2008 level of political support. For example, the Pew Research “estimate of the national popular vote is Obama 50% and Romney 47%, when the undecided vote is allocated.”

If Obama manages to win, he may do so by bucking the trend by winning with less public support in his 2nd election than he did in his first. He may then have even less political capital in his second term.

How is this outcome (or even losing) possible? Carter and Bush (Senior) certainly had their issues but what is Obama’s problem? Is it the continued economic uncertainty? While the economy has improved since 2008 it clearly remains challenging for many Americans. Is it failed promises? Perhaps. A strong opponent? Seems unlikely, even if Romney has surprised some people. Unrealistic expectation? Probably.

A look at Presidential Approval indicates that Obama has struggled throughout his presidency. Obama has an approval rating in the high 60s when he was about to take office but by August 2009 (less than a year into his Presidency), his approval rating was hovering near 50%. Eventually, it would fall to a low around 38% in August of 2011. While Obama has returned to an approval rating of around 50%, he is clearly divisive.

Obama became President in the midst of economic gloom riding a message of hope when faith in the political class was clearly more shaken than ever before. He does not wear the economic failure of either Bush Presidents (Sr. in 1992 or Jr. in 2008 when he left) but his Presidency, like the American economy has stalled.

As the first African American to hold the office of the President, Obama’s legacy is, perhaps, secured in history. If he loses this election, we may wonder what else will be written about him. If Obama-care does not survive under a Republican administration, what else do we hang on this President? It seems like he had the title of two-term President locked up but his incumbent advantage may have been more illusory than real. Perceived Republican weakness rather than strong support for Obama may have driven this expectation.

Losing now to a vulnerable and, perhaps even weak, contender will tarnish his accomplishments further. Fairly or not, Presidents are  expected to win twice.

September 8, 2012


Beating a Dead Horse: Election Polling is Flawed

Just this week, we got another pseudo apologetic letter about why election polling is sometimes (perhaps even often) inaccurate. This time it was from Peter Loewen (an Assistant Professor of UofT). His article in the Ottawa Citizen is here but, it like many similiar pieces completely misses the point.

Pollsters are in a unique an perilous position in the days leading up to an election because as much as they want o believe it it not true; voters are allowed to change their mind.

Peter’s article which is largely about the Quebec provincial election is a disservice to the industry. His focus is on the problem of getting a good sample. In his own words:

Pollsters simply do not know enough about who responds to polls via some media, who replies through others, and what kinds of people ignore polling requests entirely. The problem isn’t getting sample, it’s getting good sample. (Peter Loewen)

Does sample matter? Yes. Is their a challenge with getting a representative sample given response rates and new technology? Yes. These are not even close to being the issues that challenge pollsters in being more accurate about the vote. They are simplistic and naive assertions which miss the fundamental challenges of election polling.

1. Not Everyone Votes

Public opinion polls at there best are uncomplicated because everyone has an opinion or a non opinion. When it comes to elections, what people say is their vote choice (or non choice) is a spur of the moment decision that is divorced from actually voting. While pollsters can expect that most people who express an opinion will actually vote, we know that there is a gap when 75% give us a vote preference but only 50% or so turnout. What we lack is a guaranteed method of identifying those who will actually vote. The best sample in the world won’t address the gap between those who have a preference and those who will vote.

2. Voters can Change their Mind

I am sure that many would prefer that people did not change their mind but they do. And, given the importance of a democratic franchise, perhaps we should applaud those ballot box reconsiderations. In either case, voters are allowed to be coy — about their intention — and ultimately deceive pollsters. Good pollsters and good media know when this is a possibility.

So, sure, the sample matters but in reality, it could easily be argued that many of failures of polling are simply a reflection of a misguided focus on predicting a future election outcomes (the result) and not enough focus on the meaning inherent in what people tell us. There is very little science that can tell us how people (rather than atoms) will behave in a particular situation. Perhaps it is time to stop expecting polling to do that.