A good scientific question contains two main elements:
- Clear definitions
- Measurability.
This is something you probably know inherently, but just never really sat down and put it into words.
What constitutes a clear definition? In order to ask a good scientific question, each aspect of it needs to be well defined. A great example of this can be found in our article “Why is Pluto not a Planet?” As we discussed in that article, until 2006 the term planet wasn’t very well defined and had evolved over the years to mean a lot of different things. It took a group of scientists collaborating to determine the official scientific definition of a plant before the scientific community could start making scientific determinations as to which celestial bodies were actually planets.
Next, to have a good scientific question, it needs to be measurable. Being able to measure things is a huge part of science. If something is not measurable then it’s subjective and the answer is something that can change from person to person. For example: something tasting good; the concept of tasting good is not really measurable, and it changes from person to person and therefore hard to determine scientifically.
Now that we know what constitutes a good scientific question, let’s look at a good example of a bad question: Who makes the best Coffee?
Immediately, we can see issues with clear definitions. For example, how do you define best? Who do you mean by the word who? Is that going to include the solo coffee bean farmer in the middle of nowhere, that has been making coffee for generations in his kitchen or just big retailers in a specific country? What type of coffee are we talking about? One cup of steaming black coffee? Iced coffees? Cappuccinos? Espressos?
And not only do we not know what we want to measure, we have nothing that can be definitively measured. For example, how would you measure what is best?
We scientists often try to take non-scientific questions and give them scientific answers by nailing down the terms and definitions and figuring out how to measure them. For example, let’s say I tried to scientifically answer the question of “who makes the best coffee?”. First, I would set up a very large study across the US. As part of the study, I would poll several hundred thousand people for demographic information such as age, location, income, how much coffee they drink per day and then ask the participants to rate the basic coffee from major retailers on a scale of one to one hundred. I would then give that data to a statistician who could take that data, and come up with a definitive answer to the following question:
What basic coffee is statistically the highest rated coffee out of retailers A, B, and C within the United States, controlled for the different demographic aspects we polled for, as measured at a specific date.
As you can see, our scientific question is much different than our original question. But this gives you an example of how a scientist might be trying to answer the original non-scientific question.
We also see the converse of this when news programs and social media take scientific answers and twist them to fit non-scientific questions or hyperbolize them. For example, with our scientific coffee question, if our scientific answer was published, you could easily see news stories claiming that “Scientists have determined the best coffee ever.” This is something that happens in the media a lot and unfortunately gets pushed well beyond the truth. An article will be published giving an answer to some very specific scientific question and then the media takes it out of context and says something much different than what was originally proposed. Here is a great example of this phenomenon. Several researchers published an article in the Journal of Physiology entitled Improvements in skeletal muscle strength and cardiac function induced by resveratrol during exercise training contribute to enhanced exercise performance in rats.
The study showed that mice showed higher endurance during a 12-week program when being supplemented with a chemical called resveratrol. Resveratrol is a chemical that can be found in red wine. As you can see and if you read the whole journal article, the researchers performed a scientific study, and gave a scientific answer: Mice show increased endurance for a specific exercise regiment when being supplemented with resveratrol. Simple enough. The problem is that the news agencies, and articles written online, twisted that very specific scientific question to say something much different:
“A Glass Of Red Wine Is The Equivalent To An Hour At The Gym, Says New Study”
Link to Huffington Post Article
This statement was reported repeatedly all over the internet, Facebook, and even major news programs which is somewhat scary because that was not the actual answer to the scientific question posed.
Once you know what to look for in terms of a good scientific question, and therefore a good scientific answer, you can start to weed out a lot of things that are just plain false. The moment you read that headline, you could pick out the terms equivalent, red wine, and ‘an hour at the gym’, all of which are not well-defined terms. You’d also pick up that equivalent would not be something that would be easily measured. Based on these two things your BS meter should start ringing like crazy.
And if you have questions on an article you read, don’t forget to ask a scientist.
Further info:
Great clip by John Oliver on Scientific Studies
One of the authors of the “Red wine study” comments on the news.
Citation for Journal:
Dolinsky, V. W., Jones, K. E., Sidhu, R. S., Haykowsky, M., Czubryt, M. P., Gordon, T. and Dyck, J. R. B. (2012), Improvements in skeletal muscle strength and cardiac function induced by resveratrol during exercise training contribute to enhanced exercise performance in rats. The Journal of Physiology, 590: 2783–2799. doi:10.1113/jphysiol.2012.230490