As every driver knows, every road has a tricky section. Online surveyors will tell you the same thing: every survey has some tricky parts. Anticipating and navigating around those common mistakes sections is what makes the difference between a good survey and a not-so-good one.
So, what are those common online survey mistakes?
Common Online Survey Mistakes
The starting point of anticipating common online surveys mistakes is to master the survey process. In brief, a survey involves four major stages, each of which may be further broken into smaller parts. These are the survey planning stage, the survey design stage, the implementation or conducting stage, and the survey data analysis stage. Each of the stages has its own dynamics and its own potential to negatively affect the results. You can get a detailed explanation of that in our article Creating Simple Surveys Has Never Been Easier.
A survey could therefore fail because one or more of those stages have been mishandled. But the common mistakes are rarely about the big picture. Usually, surveyors will faithfully follow all the stages but still get caught by a small detail that somehow slips through their scrutiny. Here then is our list of mistakes to look out for.
Population Specification Error
This is a knowledge or familiarity problem. It happens when you do what seems like perfect sampling, except that you have missed a critical segment of the population that affects the information or data that you are looking for. You will probably get results that shed some light into the issue you are investigating, but they don’t provide the full picture. You might also end up with totally misleading conclusions. The error is most common when surveying household products or services. Think about it: one of the parents decides only for some the products that are bought by the family, and even then the choice is often influenced by their kids. So whom do you survey?
Trick to deal with population specification error: an easy trick would be to check out the method and target of sampling done for similar surveys in your area of interest. That way, you can make sure you’re not forgetting a critical segment and also that your sampling method is correct.
Sample Frame Error
This is a more serious sampling error. It occurs when you unknowingly sample the wrong segment of the population, usually because you have missed a new trend or shift in social dynamics. For instance, you may be looking to establish how popular a certain health drink is and therefore you sample equal numbers of males and females between ages 18 and 55 with a certain level of income and residence parameters. But, unknown to you, retirees have taken to the drink in a big way, although they usually never purchase it themselves but instead send for it. Your results will be at best misleading and at worst totally wrong, yet you in fact did proper sampling with the information you had.
Possibly the most infamous sampling error is that which occurred in the American 1936 pre-election polls when the leading media house of the day used a sample drawn from telephone directories and club memberships, only to discover later that the economic depression had distorted the representativeness of those sources. You can read about that incredible survey failure in our article The Gallup Opinion Poll: Surveys That Sampled 50,000 Instead of 10 Million and Prevailed.
Trick to deal with sample frame error: take some serious time to study the industry or area of interest of your survey before sampling. And also here, you can double-check by checking out other surveys that are similar in nature as yours.
This is a processing error. It is the extent to which your survey data varies from what it should be because of mistakes in how the survey was conducted. Usually, the problem is poor question wording, wrong assumption behind the questions, or wrongly focused questions. It is essentially a questionnaire design problem that can be surprisingly difficult to detect, because it’s just difficult to know when you have mishandled the questions if the respondents have returned lots of interesting data.
Trick to deal with sample frame error: read up on questionnaire design, particularly the tips on the types and presentation of questions. Our articles How To Write a Short and Effective Survey and Tips and Tricks: How to Best Construct Survey Questions offer more details on that.
This is usually a sampling problem. It happens when the majority of those contacted in a survey have a special interest or strong views in the survey topic. Such groups or population segments will often have a perspective that is heavily tilted or biased, and will subsequently tilt the final survey results. The problem is most common in political and religious topics where the risk is always high that the people who respond to the survey might be supporters of one side or those with a vested interest in the outcome of the survey.
Trick to deal with selection error: by really knowing your respondents, you can modify your sampling method accordingly. This way, you can increase the response rate of even those that aren’t too keen on responding. Read more about knowing your respondents here.
this is a response problem that can at first seem inconsequential. It is normal for some targeted respondents not to respond, which makes it difficult to tell what their responses would have been. The problem occurs when most of those who fail to respond hold a perspective that is different from the majority of those who responded. As a result, the final data is tilted to the view of those who responded. The effect can be pretty serious, particularly if major financial investments are subsequently based on the survey results.
Trick to deal with selection error: encourage high response rates by keeping the questionnaire short, and either conduct a follow-up survey or contact those who did not respond through alternative means. Read more about increasing your response rate in our articles Tips and Tricks: 5 Easy Ways To Increase Your Response Rate and What To Do For a Higher Response Rate.
Is anything still unclear for you? Do not hesitate to let us know in the comments! Otherwise, why don't you head on over to our sign up page to get started?
Photo credit: Fragglehound (thanks, Fragglehound)
P.S. If you liked this article, you might also like 7 Tips To Minimize Response Bias, 10 Common Mistakes To Avoid When Crafting Online Surveys, and Random Sampling For Online Surveys.
Mar 25, 2014