The Effect of Sample Size On a Survey's Results

The Effect of Sample Size On a Survey's Results

SURVEY MYTHS DEBUNKED - The More People Surveyed the Better the Results

Is it true that the more people you survey the better the results? Does it matter how many email inboxes the survey questions land in? Put differently: is sample size really such big a deal?

It’s a question that’s tied to a more basic question: how reliable are online surveys anyway? To some people surveys are no more than guesswork, because the only way to really know what people think, feel, or want is to ask everyone, and not just a few. To others, the conduct of surveys is a science that – if done right – will have enough statistical significance to be correct.

The truth – as with most things in life – is in neither extreme. Good surveys are a science and not just guesswork, but they can easily be conducted wrong. And no, the correctness of survey results lies not mostly in the absolute number of people interviewed.
What’s more important is your method of survey sampling.

What Is a Survey Sample?

Actually, let’s start from the beginning: what is a survey? A survey is way of finding out what a large group of people – like an entire country, several countries put together, or a just certain group of people who use a certain product or service – feel, want, think, or prefer about something.

The best way to do this is, of course, to ask everyone in that particular group or country. That is what governments do when they take a population census. They go to every home, count every body, record their ages and gender and economic situation, and ask a whole bunch of other questions.
Basically, a census is the mother of all surveys.

But if you are a business simply looking to know what people feel, want, think, or prefer about your company, product, or service, taking a census would be way too much. It would cost a lot of money, it would take forever, and, if you really think about it, it would be unnecessary.
Because why ask everybody if you could figure out the answers by asking just a few?

That is where survey sampling comes in. Sampling is the selection of a small group from a much bigger group in such a way that you will get a fair representation of the bigger group. The logic is that there are similarities and shared characteristics in every large group. So, if you can just figure out how to select a small group that has the same characteristic as the entire group, you could tell what everyone – or at least most of the people – feel, want, think, or prefer.

How Survey Sampling Is Done

Sampling can be broken down into two main types, with each having some sub methods:

  1. Probability Sampling or Scientific Survey Sampling

    The first is what the statisticians call probability sampling. That means that there is a certain mathematical motivation behind it, and it's therefore the best and most accurate way of survey sampling. You’ll go for this type if you want your sample to be as close to the population as possible. In this class there are four major methods of sampling techniques:

    • Simple random sampling: this is when you choose the people to survey randomly without any order or preference, like taking an email list and choosing 10% of them in no specific order and without any preference.
    • Systematic Random Sampling: this is a more organized version of random sampling, where you take an email list and choose, for example, every 3rd person on the list.
    • Stratified sampling: this is when you first divide the target population into smaller groups, like separating email addresses according to how much each one of them earns or their gender or education level. Then in each group you systematically choose whom to talk to, for example every 5th person on the list.
    • Cluster sampling: here you divide the target population into groups that seem natural, which almost always means based on geographical spread. After having done that, you can perform a simple random sampling within each cluster.

  2. Non-probability Sampling or Convenient Sampling

    The second type of survey sampling is non-probability sampling, which has less of a mathematical basis. It is a method that is less reliable than probability sampling, and it's often done for the convenience of dealing only with the people that are easily accessible or that require the least expenses. With this kind of sampling you're not looking for the exact truth, but just a decent approximation of the truth. It can be useful as a way of testing or gauging the market in a very simple way.

    There are three methods of this kind of survey sampling:

    • Judgment sampling: this is when the person conducting the survey uses personal judgment or experience to choose the people to interview, like taking a list of customers and choosing only the ones you consider serious or better informed, or standing outside a supermarket and choosing to talk only to the people that look respectable.
    • Quota sampling: in this method you first classify the group you want to talk to into smaller groups according to gender, income, or education levels or some other criteria, just like in stratified sampling. Then you choose whom to talk to according to your judgment of which ones are serious or more representative for everyone else, or simply who is easier and least costly to reach.
    • Snowball sampling: This one is a bit special. It is done by asking people whom you have just surveyed to recommend or refer you to other people they think you should talk to. It’s usually the economical way of sampling a group that is difficult to locate, like people with a certain illness or uncommon hobby, preference, or interest, as those people will usually know other people with this particular characteristic.

So, Does Survey Sample Size Matter?

In the end, it depends a lot on the population you want to survey. Then, once a method of survey sampling is selected and conducted in a good way, the number of people that needs to be surveyed becomes clearer.

Think of it this way: if the size of a certain population is huge, but most of the people within that population have the same characteristics (like income, education, religion, culture), the number of people you will need to sample before you’ll have a fair representation of the entire population can be quite small. Yet if the overall population size is small, but has a lot of diversity within their characteristics, the number of people you will need to survey will be larger. Even then, however, there comes a point at which it just doesn't add much value to add another 100 people to your sample.

So, to sum up: it is not just about the absolute number of people surveyed.
What's more important, is that you map your target population correctly, and then sample accordingly.

Is there something about surveys that you think is confusing or often done wrong? Give us a shout and we will do our best to help clear it up!

Photo credit: Max OrHai (thanks, Max)

Jonah Njonge
Jan 10, 2014
blog comments powered by Disqus