Survey research is a challenging science

  • National Newswatch

The basic theory is that a randomly selected sample of a population will more or less resemble that population. Ask 1000 random Canadians a question and their collective answer is generalizable to the overall attitudes of Canadians.But often we see polls that don't necessarily follow that scientific methodology.One of the early failures of polling was in 1936. The Literary Digest had conducted a straw poll of voters in US Presidential elections for twenty years, and every time they correctly predicted the winner.The 1936 poll showed Republican Alf Landon overwhelmingly defeating Franklin Roosevelt. But when the votes were counted, Landon won just two states and Roosevelt was re-elected in one of the largest landslides in history.At issue was how the magazine conducted their survey. While they asked 10 million Americans to respond, and 2.4 million people did, the people they asked were much wealthier than the average American.In the science of polling, there was a participation bias where the participants disproportionately possess certain traits that affect the outcome.George Gallup made his reputation in that election, calling the actual result within 1%, in part because his much smaller sample was randomly selected and demographically representative.That random sample is why you see a margin of error associated with most scientific surveys. The larger the random sample of people, the more that sample should resemble the overall population, and the lower the margin of error.Elections generate a significant media opportunity for polling organizations. But the cost of random sample telephone surveys is significant. As a result, we see a large number of less expensive internet surveys.This technique sees surveys administered to people who have opted in to be questioned, and then the demographics controlled to resemble the general population.Because there is no random selection, there is no margin of error on a survey like this.The Pew Institute found that internet based surveys differ from telephone based surveys in three important ways.People expressed more negative views of politicians in web surveys than in phone surveys.People who take web surveys are less likely to admit certain demographic groups face discrimination.People who take web surveys are less likely to report they are happy with their family or social life.When looking at web based surveys, expect you are going to see differences between phone and internet when some politicians are well known and others are not, or when discrimination is a significant issue.Sounds a bit like the Ontario election…In the last two weeks, the Ontario election has had exactly one survey conducted by random telephone sample, undertaken by Nanos for the Ontario Real Estate Association.It reports the race at PC 42, Liberal 31, NDP 21 and the Greens at 5.However, because it was conducted as part of a report on housing for an association, media and poll aggregators are ignoring this survey.This could be a mistake.With just one random telephone poll in the market currently, it is worth noting that survey methodology could be a major factor in tracking the race.Keep an eye to see if telephone surveys have large differences in the negatives voters report for politicians or how they react to discrimination, and their support for parties that are taking those positions.Lise Jolicoeur is a director at StrategyCorp. Previously she served as Executive Director of Communications to Premier Kathleen Wynne.