0 courses selected
Well, is size everything? (10 Mar 08)
When it comes to surveys, the answer, infuriatingly is: it depends. But you’d better make sure you know why – and when
Recently a local authority who had sent a postal questionnaire to all its 30,000 households contacted us.
They were worried that they had wasted money in sending it to so many people, particularly as they only got about 200 back.
When I asked them why they hadn’t just taken a sample, they replied that they didn’t know how. In any case they felt on safer ground using a census approach by sending it to everybody.
Sampling is a tricky business that you shouldn't attempt if you don't understand it. But I am always amazed that people don’t buy in a day’s help to get it right. Particularly when the savings can be so great.
The survey must have cost tens of thousands of pounds in direct costs alone, plus the staff costs. With the help of a chartered statistician for around £700 for a day’s guidance a fraction of the 30,000 could have been surveyed and the results would have been just as representative.
But how big should your sample be? I thought it might be worthwhile reproducing a section from Effective Research, which Kathleen Greaves and I wrote for the Housing Corporation some years ago. I hope you find it useful.
“(Sampling size) ..is probably the sampling topic about which the most confusion and uncertainty exists, and where some of the commonest errors are made.
Decisions on sample size are crucial for making sure that the results are reliable. Very small samples will not give results which can be relied upon, but on the other hand researchers often make the mistake of choosing a sample which is larger than needed.
Sample size is one of the key determinants of cost e.g. a large sample for face-to-face interviews may require weeks of interviewer time and data input, and it is therefore important not to select too large a sample. In addition a decision to stop data collection before the full sample is interviewed may lead to biased results. Organisations may also need to consider whether the scope of any study should be restricted in order to reduce costs.
Non-response is dealt with more fully below. In calculating sample size, it is essential to allow for the likely rate of non-response i.e. The proportion of people who do not take part e.g. if the expected non-response rate is 30%, the sample size would be increased to yield the correct number of completed questionnaires.
Main factors of importance in sample size.
Most people believe that the proportion of the population which is sampled (the sample fraction) is the most important figure to be taken into account. E.g. that a sample of 20% of the population is much more reliable than say a sample of 5%.
In fact, this is far less important than the actual size of the sample e.g. many of the large national surveys rely on samples of around 10,000 and for smaller surveys, samples of 500 or less are often sufficient. The only case where the sample fraction is important is where the population is quite small, e.g. for a population of 200, it is advisable to have a sampling fraction of about 50% i.e.100.
If only simple analyses of the data are required.
The percentage of all respondents who are satisfied with particular aspects of the repairs service, samples of 200 to 300 may be sufficient. The key determining larger sample sizes is whether there is a need to break the results down into sub-groups. Most housing surveys are used to examine sub-groups within the main sample. E.g. people living in different types of property, the area office in which they live, different age groups, ethnic origins, and employment status are some of the most common.
Analysis of the results will make it possible to examine differences between the groups e.g. in a survey of demand for low cost home ownership among tenants, you will probably want to identify the interest and views of different groups such as families with higher income levels, or younger tenants. In these cases, a general rule is to make sure at least 100 of each sub-groups are selected. This can, of course, only be done if prior information exists about sub-groups in the study population.
In some cases , the numbers in the sub-groups of interest may be very small; there may be very few tenants in properties adapted for the disabled. Applying a quite small sampling fraction across the board, say 5%, may be sufficient for the overall views of everyone, but may result in only a tiny number of responses from the group of interest.
In such cases, then you might choose to make them the subject of a short separate study. Alternatively a larger proportion of the group of interest may be chosen (oversampling), which brings their number up to the desired level. In this case, a weighting procedure must be applied at the data analysis stage to make sure that the group is not over-represented in the overall results of the survey.
If you need someone to help you conduct your own survey give me a ring. We can happily suggest some good freelance survey consultants. They don’t have to cost the earth and you needn’t use them to do the whole survey. They could just help you draw your sample, cast a critical eye over your questionnaire or simply advise you on how to get started. They could save you a lot of heartache and help you do a more professional survey.
And why not attend our one day course, An introduction to sampling for surveys or How to make sense of statistics and data analysis– an introduction? Bring a colleague as we give you a discount of £25 for the second booking.
Like to read more or make a comment? Log in or register below
Why not join the discussion!
Or even better still offer your own advice and tell us about things that others can learn from.
We moderate comments lightly so bear with us and we'll get your thoughts listed as soon as we can.
You must be logged in to post comments.
Not registered yet? Simply fill in the box below.
