Survey Research: A Summary of Best Practices

December 31, 2004
Document

Ethics Resource Center 2004
Leslie Altizer

Surveying, whether for marketing, social sciences or public opinion, is the most efficient method for collecting information about a large group of people. A researcher is able to collect the opinions, perceptions and observations of a small, representative subset of a population in order to generalize to the whole. However, this capability is not to be taken lightly. Many dangers exist if a researcher opts to conduct a survey irresponsibly and without rigor. Luckily, many organizations and associations have developed sturdy guidelines that, if followed conscientiously, will result in a sound study.

The Council of American Survey Research Organizations (CASRO), the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP) are just some of the groups who publish such guidelines. This paper will briefly discuss each of the major principles that a researcher should follow when conducting a survey based upon the guidelines of CASRO, AAPOR and NCPP. For an in-depth explanation of these guidelines, please visit www.casro.org, www.aapor.org, and www.ncpp.org.

Problem Definition/Goal Setting

Before beginning to develop a survey, a researcher must conduct a "needs analysis" and obtain background information. The researcher should always get information regarding the client and the level of resources the client has at stake and determine the purpose of the survey, or what decisions will be made based on the results. Understanding this purpose will allow the researcher and the client to determine the goals and objectives of the survey. Goals should be clear and concise so that they can be directly associated with the survey results. Specific goals reduce the risk of the data and results being manipulated, or accused of being manipulated, to serve a hidden agenda. The appearance of evil or impropriety is enough to discredit findings.

While determining the goals and objectives of a survey, it may become evident that an alternative method for collecting the data is necessary. It is possible that other methods, such as focus groups or one-on-one interviews, would be a more useful method for gathering the requested information. If a researcher feels that the data would not be best obtained through a survey, it is his or her responsibility to inform the client of this and offer alternatives.

Selecting a Sample

Perhaps the most important step in conducting a survey is selecting a sample that will adequately represent the population of interest. If this is done incorrectly, a survey and the work that is expended in the research is rendered meaningless. There are various sampling designs (i.e., convenience, random, stratified-random, panels, etc) that can be selected, but a detailed description of each is beyond the scope of this paper.

The important elements to a dependable sample are to ensure that the correct population is being sampled and that all members of that sample are identified and have an equal opportunity of being selected. When the results of the survey are being reported, it is imperative that the rationale for the sampling design be disclosed. Disclosure will involve describing the "universe" and explaining why and how the sample was drawn.

Questionnaire Design and Testing

The research goals should always direct how survey questions are asked. Whether the data to be collected is quantitative or qualitative, the survey instrument must be designed so that it obtains objective, unbiased results. The following are some basic points to remember when building questionnaire, though the list is not exhaustive.

  • Questions should not be leading.
  • Questions should only ask for one piece of information.
  • Questions need to be relevant to the topic of interest.
  • Questions and response categories should be suitably designed and coded for the necessary statistical analyses.
  • Questions should not be threatening to the participants.
  • Questions should use language appropriate to the participants.
  • Questions should be organized in a logical manner.

Once the perfect survey instrument has been designed, the researcher must pretest the questions and procedures to reduce error and identify problems. It is impossible to forecast all the potential biases that await a survey once it is administered to a sample. Pilot testing the instrument with a small sample will reduce the sampling and response error once the survey is given to the true sample. Money and time should be budgeted into every survey effort specifically for a pretest.

Data Collection

Once the pretest has been conducted and necessary alterations made, there are a few final steps before the researcher actually sends the survey into the mainstream. If the survey is going to be done via the telephone, careful training on interviewing techniques should be incorporated. Among other things, the interviewers should be skilled at making initial contacts, dealing with difficult respondents and ensuring confidentiality, and have familiarity with the survey development and implementation process. If interviewers are not trained properly, their unintentional actions or words can impact how respondents react and answer the questions, thus resulting in unnecessary error. Also, just because the interviewers have been trained should not mean they are left to their own devices. A researcher should consider sitting-in on random interviews to ensure proper implementation.

Another important consideration is the treatment of human subjects. Ensuring confidentiality or anonymity is of vital importance. Participants need to feel they have the ability to speak freely without the fear that answers will be traced back to them. This is especially important when conducting an organizational survey or when the topic of the survey is sensitive or controversial in nature. A careful script must be written that can either be read to the participants or attached to the survey. The script should explain the purpose of the survey, how they were selected to participate, that it is voluntary and their confidentiality or anonymity will be protected.

Finally, researchers should set-up some form of helpline or Frequently Asked Questions (FAQs) in case participants want to voice concerns or have questions. Depending on the nature of the survey, the researchers can set-up an e-mail address or toll-free phone line that participants can utilize.

Data Processing

Once the data has been collected, the researcher must begin analyzing the data. No matter what statistical techniques are employed, a researcher should base the analyses on the previously determined research objectives and goals. There are some basic statistical steps that a researcher must do before any analysis begins. Some of these include setting a specific significance level and developing some sort of analysis plan that prevents "mining" of the data.

During the designing of the survey instrument, the researcher should have coded the questions with specific statistical analyses in mind. It is imperative that only the proper analyses be run with the data so the results will be valid and reliable.

Report Writing

The final phase of the survey process is reporting the findings. The major elements of reporting the results include (this list is not exhaustive):

  • disclosure of client, research company and dates of data collection;
  • purpose of the research;
  • the methodology employed;
  • definition of the population and how the sample was drawn;
  • sample size;
  • response rate;
  • any weighting of the data that was done;
  • discussion of the precision of findings, including the statistical techniques;
  • copies of the questionnaire, confidentiality agreement and any other document provided to participants.

The goal of this paper was to briefly overview the steps that should be followed in order to conduct a thorough and valid survey research study. Readers who have further questions are again encouraged to refer to the AAPOR, CASRO or NCPP guidelines, or contact Leslie Altizer, Research Analyst with the Ethics Resource Center (leslie@ethics.org).