Experiments are often designed to take place in a controlled setting, in order to reduce the number of
potential unrelated variables and possible biases that might affect the results. Some possible problems
might include: researchers knowing which participants received particular treatments; a particular
circumstance or condition, not factored into the study, that may impact the results (e.g., other
medications that a participant may be taking), or not including an experimental control group. However,
when experiments are designed correctly, difference in responses, found when the groups are compared,
allow the researchers to conclude that there is a cause and effect relationship. No matter what the study,
it must be designed so that the original questions can be answered in a credible way.
Gathering Data
Once a research plan (whether descriptive or experimental) has been designed, the subjects must be
selected, and data must be gathered. This stage of the research process is essential to generating
meaningful data. The ways in which data are collected vary with the type of study. In experimental
designs, the data should be collected in the most controlled manner possible, in order to reduce the
possibility of generating contaminated results. Some experiments require more strenuous procedures
than others. When gathering data on people’s perceptions of a new business marketing strategy or data
concerning the effectiveness of a new teaching strategy, the consequences of inaccurate results are not as
critical as they would be in a medical study. Therefore, in low-stakes experiments, it is sometimes
preferable to use less robust data gathering procedures in order to save time and money.
Selecting a Useful Sample
In analytics, as with computer programming, garbage in results in garbage out. If subjects are
improperly chosen, for example by giving some more of a chance to be selected than others, the results
will be unreliable and not useful for making decisions. For example, John is researching the attitudes of
individuals about a possible new tax. John stands in front of a local grocery store and asks passers-by to
share their thoughts and attitudes. The problem with that is that John will only get the attitudes of a)
individuals who shop at that grocery store; b) on that specific day; c) at that specific time; d) and who
actually chose to participate. Because of his limited selection process, the subjects in his survey are not
representative of the entire population of the town. Likewise, John design an online survey and ask
people to input their feedback on the new tax. However, only people who are aware of the website, have
access to the Internet, and choose to participate will provide data. Characteristically, only people with
the strongest attitudes are likely to participate. Again, these the participants would not be representative
of everyone in the town.
In order to avoid such selection bias, it is necessary to select the sample randomly, using some type of
process that gives everyone in the population the same statistical opportunity to be chosen. There are
various methods for randomly selecting subjects in order to get valid and useable results.
Avoiding Bias in a Data Set
If you were conducting a phone survey on political voting preferences, and you made your calls to
people’s land lines at home between the hours of 8:00 a.m. and 4:00 p.m., you would fail to get feedback
from individuals who work at that time. Perhaps those who work during those hours have different
preferences than those at home during those hours. For example, more business owners may be at home
and express voting preferences for something completely different than members of the working class.
Surveys that are poorly designed may be too lengthy, resulting in some participants quitting before they
finish. Participants may not be completely honest if the questions are too personal. If the list of choices
is too limited, the survey will not be able to capture valuable data that people would have provided.
Many things can render survey data invalid.