April 22, 2013


The Fragile Nature of Consumer Confidence

The impression one gets from a casual following of economic news is that Canada is doing well but not outstanding. There are concerns about Europe and the U.S. but these seem divorced from the day-to-day lives of most Canadians. Nevertheless, consumer confidence has dropped quite precipitously over the past three months. According to TNS Canada, […]

Continue reading...

April 15, 2013


Questions I Hate on Surveys: A Continuing Series

I spend a lot of time on the Internet, as you probably do as well. It is there in the background when I am working, it is in my pocket when I am walking, it is playing music in the kitchen as I prepare dinner, and it is sending me alarms and notices. When I […]

Continue reading...

March 12, 2013

1 Comment

An Historic Evolution of Public Opinion: A Look Back at Same-Sex Marriage in Canada

Public opinion change is not unheard of but when it comes to fundamental beliefs and values, we expect change to take place slowly, if at all. The evolution of Canadian beliefs about same-sex marriage provide an interesting example of dramatic changes that both presupposed and reacted to court decisions.

 Courts played a key role in Canada but public opinion was not an idle bystander.

This is telling as Americans look to an upcoming Supreme Court decision about same-sex marriage that will no doubt shape the political discourse there.

Court decisions were certainly critical points in the evolution of public opinion on the issue. For timelines of the issue see the CBC website. Notably, courts ruled in the early 90s that discrimination based on sexual orientation was protected, that same-sex couples have the same  benefits and obligations as opposite-sex common law couples (Supreme Court, 1999), and in 2002 the Ontario Superior Court rules that prohibiting same-sex marriage is unconstitutional. Ruling on same-sex marriage in other provinces followed until eventually the Parliament of Canada passed a law defining marriage to include same-sex relationships in 2005.

Throughout the period of the 1990s Court decisions were in line with and reflected a growing tolerance toward homosexuality. Attitudes toward Gays and LesbiansThankfully, the Canadian Election Study (CES) and Environics both tracked the evolution. Starting in 1987 Environics asked as part of its Focus Canada study, Canadian attitudes toward homosexuality. In 1987, 10% indicated that they approved of homosexuality (there were many unsures (30%) this year and in 1996). By 2004, this number reached 47%. Even setting aside the very low year of 1987, the increase from to 47% is remarkable. In a different approach, the Canadian Election Study asked respondents to rate homosexuals (Gays and Lesbians after 1993) on a scale from 0 to 100. The average rating for each survey rose from 45.5 to 73 out of 100 in 2011.The outsider nature of the community has broken down substantially in the last decade, particularly after 2004 (between 2004 and 2011 the average rating has increased much more than it did between 1993 and 2004).

Although history seems like inevitable march toward granting and accepting same-sex marriage, in 1999 a federal vote maintained the traditional definition of opposite sex relationships. While courts and Parliament were extending the same rights, the actual inclusion into the institution of marriage was being denied. Public opinion was divided and offered little in specific guidance.

The chart below shows polling on the question over time from four polling firms. There were many other polls but often the questions were only asked once and this provides a very interesting as it is.

Gay Marriage-Trends

The CES surveys, which cover the longest period, show agreement that Gays and Lesbians should be allowed to marry rises significantly between the 1997 and 2000 elections. Note that in this period, the Supreme Court had ruled that same-sex couples have the same benefits and obligations as opposite-sex ones. The question of marriage was definitely on the agenda. Nevertheless, just 50% supported marriage at this time.

As the issue heated up, more polling was taking place and polls showed relatively little movement in attitudes. Most trend lines were not moving dramatically in either direction but overall most were showing support for same-sex marriage hovering around 50% depending on whether respondents were explicitly given an unsure or neither category (this is why the Ekos trend line is lower). All trend lines tend to be more positive once Parliament acted in a manner consistent with the court rulings.

One reading of the trend is that public opinion was divided but leaning toward accepting same-sex marriage but this misses a key element of public opinion at the time — the overall acceptance of same-sex relationships.

The questions in the figure above show the issue in terms of a binary choice — same-sex marriage or not. The reality is that There were fewer staunch opponents than it may seem. A TNS survey in 2004 found that 39% would support marriage and another 35% would support legal unions for same-sex couples.

Moving forward to recognize same-sex relationships in formal-legal way (possibly including marriage) was something that Canadians were overwhelmingly onside with. Courts said it had to be civil marriage and a healthy majority of Canadians now agree.

Canadian courts played a nontrivial role in bringing about same-sex marriage but they did so largely with public support. If courts were leading they found increasingly receptive audiences for their messages.

We are fortunate that the Canadian Opinion Research Archive is around to help preserve the cultural legacy of the public opinion research that has taken place in the past 3 plus decades. I draw on some of the data here along with searches of individual company websites.

March 20, 2012

1 Comment

Does your scale have a tipping point? Questionnaire Design

Question scales come in two types: seesaws and stairs.  The main point of differentiation of the two ideal-types is the presence or absence of a tipping point. See saws have them and stairs do not.

When you are thinking about a scale, you need to ask yourself is there a point on the scale that the scale tips to the other side because this will help ensure you have the right points on the scale.

Some questions naturally have a tipping point

Consider an agree-disagree scale. Regardless of the number of points on the scale, at some point the respondent must be on one side or the other (or precariously balancing in the middle). As you move from strongly agree to strongly disagree, at some point you move from on-balance agreeing to on-balance disagreeing. A yes-no question also has a natural tipping point. Thus a seesaw

But not all question scales have tipping points because they are asking, how much of something do you have? Just like there is no tipping point (a single place in the scale where it breaks into two halves) for income, there isn’t one for many questions that capture the amount of something like concern, interest, or priority.

How concerned are you about the safety of the food you buy?

The question above would be best captured with a scale that went from “Not at all concerned” to “Very or extremely concerned”. As you can see, there is no place where you specifically tip from one side to the other. The scales act more like stairs or a ladder with artificial stepping points on an underlying dimension.

Of course, you can turn this into a seesaw scale by polarizing the poles of the scale (e.g. very unconcerned to very concerned).

Why is matters

To some extent using a seesaw versus a stair scale is a matter of preference but there is one place where it matters. When you have a stair type scale that moves up from nothing to a lot of some thing (e.g. importance, concern), then there is little place for a neither category in the middle.

Neither categories fit nicely into the middle of seesaw questions because a respondent can refuse to take sides — balancing in the middle. They do not work the same way for a stair-type scale. Respondents who are unable or unwilling to answer a question must choose a category outside of the scale for these questions such as “Don’t know”, “Not applicable”, or “No opinion”. These non-answers can’t be in the middle now but rather must be outside of the main scale.

So do you prefer scales that force a tipping point or those that allow people to place themselves on a continuum from nothing to a lot of something?

December 31, 2013


2013 Jenkins Research Blog in Review: Clearly Need More Activity in 2014

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 1,600 times in 2013. If it were a cable car, it would take about 27 trips to carry that many people.

Click here to see the complete report.

June 11, 2013


Replacing Demographic Analysis with Something Else

Sometimes I think a demographic analysis is the bane of my existence. Endless bivariate comparisons of various independent variables against the questions asked in the survey. Tedious, often of tenuous value, and, of course, subject to problems of collinearity.

Demographic characteristics are not independent of each other. Saying that older people are more likely is a half truth.

Age, for example, is not independent of many other variables. The educational attainment of those older will be less than for those who are younger because after WWII, post-secondary education became more available. So education is very generational dependent. This, of course, is just one example of a broad problem of looking at a single variable in isolation.

Individuals are better thought of in terms of their life context, which is the intersection of their age, gender, working status, educational attainment (to this point), and household income (perhaps even other things), rather than in terms of their position on any single one of these. A person with low income may have different attitudes and values if he or she is a student versus an unemployed blue collar worker, versus a senior living on a fixed income.

Can we represent where a person is in their life in terms of a few demos?

Using the 2012 Focus Canada dataset from the Environics Institute, we created 8 lifestage groups with a basic cluster analysis. Although occupational status is a major driver, which is often not one of the most reported on demographic variables, age, income and education serve to further divide the population. In fact, the two largest groups, Professionals and Blue Collar represent 57% of the population.Image

The 8 groups represent fairly well recognizable segments of the population. In one variable, we can effectively combine the effects of multiple demographic variables. Small niche groups are harder to represent but the broad groups are evident and one could still sub-divide the large groups to look at age within them.

The value of the model, however, is not in being able to create the groups but in being able to offer insights into public attitudes.

Using two questions that tap attitudes about the economy — specifically whether people are better off compared with the previous generation and whether the next generation will be better off then themselves we can see the potential of this approach. The ideal that each new generation would be better off than the previous is not universally believed. Just 52% believe they are better off than the previous generation and only 25% think the future is brighter for the next generation.

Looking at it in terms of the segments is interesting because…

  • the two retired groups have a similar perspective despite their education and income differences. This is a true age/ generational effect.
  • Professionals and Blue collar groups differ fundamentally. Blue collar workers are more likely to see the erosion in the dream of being better off then the previous generation. Is this a key insight into the challenge of the economy and the differential rewards it offers?

ImageIncome, education, age and working status are all related to attitudes but none really captures the dynamic evident here because they miss a key tension.

More work is clearly needed and while the question maybe particularly suited to a lifestage approach to the analysis most attitudes are constructed based on a life experience.

Demographic analyses are a staple and tend to be done using a series of bivariate analyses primarily because this is what our table systems spit out. So the next time you see or undertake a demographic analysis ask yourself, is this the best we can do?

More information on the lifestage groups will be available in a longer format shortly.




May 29, 2013


Generational Replacement Can be Small Part of the Story of Value Change

Not too long ago, I wrote about the dramatic change in public opinion about same-sex marriage and our natural tendency is to think of these types of changes as being a reflection, to some degree, of generational replacement. Younger people have different attitudes than the previous generations and over time they become a larger proportion of the population. Generational replacement can be an easy explanation, especially when the change involves values because we have always been taught that values  are the most enduring aspects of public opinion.

When it comes to same-sex marriage, Canada became more accepting/ more tolerant because Canadians changed their minds. Only a small percentage can be attributed to new generations entering into the adult population.

Some of the data analyzed goes back to 1993. This was a time, when less than one in five adults was a member of Generation X (born between 1966 and 1980). By 2011, Generation X accounts for 24% and a completely new generation, Y (born after 1980) represents 11% of adults. Since generation Y is very liberal on social issues, this could go a long way in explaining the trends.

Between 1993 and 2008, support for homosexual/ gay and lesbian marriage, as measured by the Canadian Election Study, rose from 37 to 64% (a massive change in level of agreement of 27 points). The chart which shows the evolution of public opinion in terms of four main generations indicates that:

Attitudes toward same-sex marriage in Canada by generation

Attitudes toward same-sex marriage in Canada by generation

  • The oldest generation, born before 1946, remains the group most offside when it comes to same-sex marriage.
  • It appears that generational replacement might be a significant part of the change as support for same-sex marriage among generation X and Y is flat or only increases marginally between 2000 and 2008.
  • Clearly both older generations dramatically changed their views on the issue. 

In fact, if the demographic makeup of Canada in 2008 was the same as it was in 1993, the support would be 61%. Replacement accounts for 3% of the change. The only qualifier is that the timing of the changes was different. Generation X was the first generation to move in a liberal direction followed by the Baby Boomers. Generation Y entered the public sphere with largely accommodating views on marriage.

Across an almost twenty year period, one would expect that generational replacement would account for much of the value change but the results here suggest that this simple explanation overlooks the significant changes in Canadian attitudes.

An examination of the other major trend line from the CES, feelings toward homosexuals/ gays and lesbians reveals similiar results. 

April 19, 2013


The Value(?) of Credentials: What do they really tell us?

Over the past quarter century, North American society  has undergone a profound education creep. This is evident in the fact that more and more students are entering university, but also in my impression of the seemingly endless number of new degree programs and industry certifications that have emerged.

While the world is no doubt a more sophisticated place than it was 20 years ago, particularly in the sciences, a good part of the expansion of credentials seems more about furthering an insider versus outsider mentality designed to insulate insiders from competition than it is about real credentials.

The past is ripe with examples of organizations that have used licenses to restrict access to professions from guilds to modern-day unions. Credentials and by extension licenses are equivalent barriers. At some level, all of these function to limit or constrain who can (or should) carry out a particular service.

Now I am not opposed to all credentials. Admittedly, I would prefer to be seen by a physician that has a license over one that does not. But the credential itself offers little more than a minimal standard. Every credential, whether it is held by a doctor, a CMA, or a market researcher, is only a statement that the person met a standard. By definition, some of those holding the certification are much better than others. Interestingly, we don’t ask for transcripts to prove our doctor of choice was top of their class, which would be an effective measure of how much confidence the granting organization has in their abilities.  Instead, we either trust that the person knows what they are doing or we look for other validation of their ability (where they work, recommendations from friends or colleagues).

When we move from life-and-death to the more mundane matters of business and service choice, the credential seems to offer even less. For most of us, I expect we  tend to assume that a person has the required certifications and make our choice based on other factors. The minimum standard is probably not what we really care about when choosing someone to cut our hair, do our taxes, or give us financial advice. We want more, even if judging that more is not easy (and sometimes requires experimentation).

If we always want more, then credentials are really just an entry ticket. They simply get one past credential-based filters in hiring and other selection processes. We win or lose on other factors. Which raises the question, do we need all of these credentials? Would the things that we really care about (e.g. previous experience, recommendations) not suffice to exclude those who would not meet the minimum credential anyway?

As journalists have found out, there are forces in North American society that are breaking down the barriers between those with and without credentials. Anyone can now be a journalists and their content speaks for itself (for better or worse). It is not clear that other jobs face the same pressure but more credentials is unlikely to be the answer.

So the next time you see a bunch of letters after someone’s name, ask yourself what they really mean. To me, they are at best a sign of personal commitment and interest not a basis for distinguishing between two people.

Richard Jenkins, Ph.D.