The Third Way

The following are the remaining steps for hypothesis testing. The following expands on some of the steps in Morgan Jones’ book, The Thinker’s Toolkit, so there are more steps here than appear in the book.

Step 5. Review, revise, and add hypothesis

After reviewing all the evidence against the hypothesis, you may have a gut feeling that one or more needs to be reworded differently, given a different slant, or even made more specific. Reword and change hypothesis as needed.

Also, new hypothesis or ideas that you want to prove popped into your head. Write those down and put a brief title for them in the column heads.

Step 6. Review, revise, and add evidence.

Some of the evidence will need to be reworded or revised, based on the first pass. Make these changes.

Add more evidence or those “wish I had this evidence” items to the matrix. Some of these will occur during the first scan of the matrix.

Step 7. Fill in the blanks or make changes related to the changed and new items from the previous two steps.

The previous steps modified and added items. It is probably easier to begin with the changed and added evidence. Afterwards, handle the remaining evidence vs. hypothesis intersections that the first part didn’t handle.

Step 7a. Review the current consistent-inconsistent values of each of the changed evidence against each hypothesis. For new evidence, enter the consistent-inconsistent value related to the evidence for each hypothesis.

Step 7b. For each changed hypothesis, review each consistent-inconsistent value that was not reviewed in step 7a. For new hypothesis, enter in the consistent-inconsistent value at each intersection with a line of evidence.

Step 8. Lightly mark out or X out any line of evidence that is consistent with all hypothesis. This evidence is not important to further culling and analyzing of the hypothesis. However, some of this evidence may be required if you are reporting on this analysis. (Hence, don’t obliterate it, making it unreadable.)

Step 9. For each hypothesis, determine if there are sufficient “inconsistent” grades to remove it from further analysis. Again, mark it out but don’t obliterate it.

Step 10. Rank the remaining hypothesis based on the number of the inconsistent marks, and the the weight or importance of evidence to which it is inconsistent. That is, some evidence has much greater importance in the overall analysis and this must be accounted for in your analysis. This ranking should be from the more likely to the least likely.

Step 11. Sometimes, further analysis of the most likely hypothesis is important. This analysis would include time-lines, causality trees, consequences, and so on. When the above steps point to a most likely hypothesis, this means that the hypothesis was not disproved. However, the most likely hypothesis was not proved either.

Further analysis can provide supporting evidence for the hypothesis using different tools. This builds confidence in that hypothesis if the analysis aligns with the hypothesis. This is like running experiments against the hypothesis to see if it holds true under different situations and exposure.

Step 12. What does your intuition or gut feeling say? The purpose of analysis is not to simply make a decision for us. It is to inform our mind and emotions, providing them with a structure and information for decision making.

As with other techniques, we may get a gut feeling part way through these steps; our mind is becoming informed and beginning the decision process. But after it is all over, it is our intuition that we must follow.

If we feel good about the analysis having found the best hypothesis, we are done. If we have the gnawing sense that this is not right, we can revisit the evidence to determine which should be revised, removed, or something added. We may realize that we are missing one more piece of vital evidence.

Likewise, we may want to reword the hypothesis. Or scrap the hypothesis, or add a new one to the list and do the hypothesis testing over.

Next Time …

Hoping to have an example worked up regarding the hypothesis testing. From work scenarios, I hope to find one regarding troubleshooting that would apply.

Since this may take longer than I hoped, I may begin a different topic and return to the hypothesis testing example when the example has been written up and completed.

Hypothesis testing uses an eight-step method. The following covers the first four steps in detail. Subsequent blogs cover the remaining four steps and provide an example.

The following is described in greater detail in Morgan Jones’ book, The Thinker’s Toolkit. It is a great book and easy to read. (I never loan out my copy, never.)

Step 1. Generate hypotheses

We have a natural inclination to jump to conclusions, so this step allows us to give in to this natural inclination. Write up hypotheses as they come to our mind (brainstorming). Hypotheses are statements that explain or give a reason. They may be statements we want to prove or disprove.

From this larger collection, some hypotheses can be combined. Of some hypotheses we may be unsure whether or not to include them. For now they can be written and set aside for the moment. Some hypotheses can be eliminated right away.

Step 2. Create the matrix

A sample, blank matrix is shown in the figure below. Each hypotheses is a column in this table. The hypotheses column heads are labeled A, B, C and so on. Write a shortened version or title for each hypothesis in each column head.

Hypothesis Testing Blank Form

Each row contains a significant item of evidence. These are numbered one, two, three and so on. At this point, we have not listed the evidence, but created places to hold our evidence.

Step 3. List the evidence.

Which of the mass of evidence is significant? Evidence that is unimportant can be ignored. Determining importance is more a matter of context for these hypotheses.

For example, the time of day or the color of the car have no meaning if we’re trying to analyze a business issue. However, if we are solving a crime, time of day and color of the car may be the most important pieces of evidence. So importance of evidence depends on what were trying to prove and disprove, and how well that evidence supports or refutes one or more hypotheses. Often your intuition or gut feeling will guide you as to importance. If undecided, write those items on a separate list that can be used later.

Another important aspect is to overcome the human tendency to satisfice. That is, we tend to stop collecting evidence when we feel an emotional closure during this process. Keep on going. This overcomes our human tendency to see only what we want to see and ignore the rest.

Any evidence of uncertain value on this first pass is put onto a separate list for safe keeping. After completing a round of reviewing all important evidence against each hypothesis, we may want to add some of these uncertain items to our matrix for the next round.

An important category of “evidence” to consider is not actually evidence. What do we wish we knew but don’t yet know? What further evidence would be helpful if we could obtain it? The answers point to further research and analysis that can shed even more light on our hypotheses. What information am I missing that could specifically disprove one or more hypotheses? Where can I find this information? How can I find it?

Step 4. Work horizontally, examining evidence against each hypothesis

Begin with the first row, an item of evidence. Compare this evidence against the first hypotheses and mark the matrix cell as to whether the evidence is consistent, inconsistent, or we just don’t know. Mark the adjacent cell based on how the evidence compares with that hypothesis. Continue until the row is complete.

Then do the same to the next row, then the next row, and so on.

The author says it is best to work in a horizontal manner. For some reason, our minds perform better analysis when holding an item of evidence in our mind and comparing that with each hypothesis. For some reason, we do a less quality job when we hold one hypothesis in mind and compare that with every item of evidence (vertical processing).

Methodical plodding across and down the matrix prevents us from jumping to conclusions and from not carefully considering all the evidence. We avoid our instinctive jumping ahead and making false judgments.

When we say that evidence is consistent with the hypothesis it can mean a few things. It may be that this item of evidence supports the hypothesis. That is, if this evidence is true the hypothesis could be true. At least the evidence does not actively disprove the hypothesis. And maybe if the hypothesis is true, it doesn’t matter if the evidence is true or false. In logic, this is similar to the “don’t care” state. A don’t care value does not invalidate the hypothesis.

We may find the evidence is inconsistent with the hypothesis. This means, if the evidence is true then a hypothesis is likely to be false. This evidence disproves a hypothesis.

The author recommends writing in the matrix. These are helpful as your analysis continues.

For next time

We will cover the final four steps in the next blog. A subsequent blog will contain an example that steps through this entire process.

What it is not–a cursory search of the Internet shows that most descriptions of hypothesis testing refer to a form of statistical analysis. This type of analysis uses probabilities, curve fitting, ignoring some data, and deals only with numeric data. There are many textbooks and resources for that type of hypothesis testing. But that is not what we discuss here.

What about us?

We have ideas, explanations, and reasons of why things happen. We may want to prove a point. These ideas, reasons, points, and so on, we will call hypotheses. Sometimes our goal is to disprove an idea (usually someone else’s).

Too often people begin throwing around hypotheses with vague and weak reasons why they are true. And others come back with different hypotheses and vague, weak reason why theirs are true and the arguments of other people are false. And round and round it goes.

Our penchant is for jumping to conclusions on the first thing that pops into our heads. And so armed with an idea and maybe one or two positive reasons why our idea is true, we vehemently defend our position.

This hypothesis testing may not definitively resolve the argument. At least it can rid the debate of half-baked, crack pot ideas. Hypothesis testing will cull out things that do not stand up against the evidence. It will show which ideas have the most merit and value based on evidence.

The Book

Morgan Jones covers hypothesis testing in detail in his book The Thinker’s Toolkit. (Buy the book; it is very good. And I don’t make a penny from it.) The author’s discussion has excellent examples to step a reader through the hypothesis testing method. If hypothesis testing seems to be of value for your decision-making, do read the book. The author does a much better job than I.

What is a hypothesis? And what do I do with it?

First, the hypothesis is a statement we put forth as true. It could simply be the reason why something happened. It could be our best guess at the situation based on the meager evidence we have in our possession.

The author credits Karl Popper with this: we cannot prove that a hypothesis is true. However, we can show that a hypothesis is false if we find evidence to the contrary.

Hypothesis testing blends deductive logic with inductive logic. It is both divergent and convergent. Many of the best decision-making tools use both divergent and convergent steps. Divergence generates ideas (hypotheses in this case). Convergence then culls out ideas, leaving the few best ones.

Socrates–hypothesis testing made the wisest man in Greece

Stretching it a little, hypothesis testing is something that Socrates practiced. Socrates love going around and urging his victims to spout out some ridiculous hypothesis. Then Socrates would proceed to question and finally prove that their idea was faulty (at best) or possibly stupid (at worst). He won many arguments and lost many friends. But he gave us the Socratic method of coming up with ideas then finding if evidence would refute them.

Hypothesis testing – the 60 second version

These steps are a high-level view and provide a good introduction to this method. Details will be put into the next post.

1. We have a bunch of evidence and come up with one or more explanations or reasons that seem to fit.
2. All this evidence and our ideas are put onto a matrix for this method. That gives the data a structure from which we can carefully examine each and every item. The matrix uses the hypothesis as one axis and the evidence as another. (Next blog posting will show the table and go into details on filling and using it.)
3. For one specific evidence, compare it to every hypothesis. Determine if the evidence is inconsistent with this hypothesis or not. The important point is to disprove, not to prove.
4. Repeat the previous step for each item of evidence.
5. At this point, some hypothesis will be sufficiently disproved and can be removed before the next round.
6. Some new evidence may be available and this can be added to the matrix. Some new hypotheses may also some to mind and those can be added.
7. In some cases, we ask ourselves, exactly what data would disprove one or more of the remaining hypothesis? How can we obtain this evidence? Where can I find it?
8. Then we repeat at step 3.
9. If we are fortunate to obtain sufficient contrary evidence, we may be left with one hypothesis. This will give us a very likely hypothesis. And sometimes we have more than one and just can’t eliminate any more hypotheses. That is OK.

The purpose of the method is to prevent ourselves from jumping to conclusions, ignoring evidence, and having incomplete analysis. We may not come up with the one and only best answer to the question. But we can weed out a lot of ideas and items that don’t fit. Structure helps us bring out our best thinking and avoid the mistakes that can make us look foolish.

Next installment we will cover the detailed step-by-step procedure.

Hypothesis testing for decision-making

First a definition: a hypothesis is a statement or judgment we believe to be true and wish to prove so. In science or mathematics, the hypothesis is the goal of the research or the thing we hope to prove. For our everyday use, hypothesis is something that we believe to be true or we want to be true. It can also be an idea that we are trying to prove. It can also be something that we wish to determine whether it is true or not.

In hypothesis testing our natural tendencies lead us far astray. In The Thinker’s Toolkit by Morgan Jones, the author gives quite a bit of discussion to this issue of hypothesis testing. He also shows how we lead ourselves astray.

As the author mentioned earlier in his book (and we repeated here), we are addicts for a reason or explanation. And when we get one, we feel such emotional release that we search no farther. We love jumping to conclusions, and we are expert jumpers. We have honed conclusion jumping to an art form.

Hypothesis testing with the correct method helps us get closer to the truth. The correct method helps us avoid our natural tendency to latch onto the first explanation that comes to our mind.

Hypothesis testing-how it shouldn’t be done

As an example of hypothesis testing gone bad, this is how almost all of us go about this method. (I am embarrassed by how often I fall into this same trap. That’s why I know this so well.)

1. Sometimes we begin with an idea and sometimes we’re just trying to search for what we feel is the truth.
2. We go out and gather some evidence
3. An explanation pops into our head and we latch onto it. After all, since we thought of it, it must be true.
4. We search for more supporting evidence. And since we know what we want, we will probably find it.
5. We make sure to amplify the importance of supporting evidence. Even if the evidence is weak, we will prop it up so our argument becomes strong in our own mind.
6. We ignore or neutralize contrary evidence. After all, since it does not fit with our explanation, it must not be true. Or it must not be that important.
7. We pronounce that we have found the truth. And, we stick with our pronouncement.

The above method would be funny if it wasn’t so often true. The consequence of a faulty method has put many an innocent person into jail. In our own lives, we miss many good things by arriving at a less desirable solution or hypothesis or goal. The stakes become much higher when a person’s innocence or guilt is decided. The stakes are even higher when resolving pollution, global warming, or unstoppable diseases.

The main failure for our own ad hoc hypothesis testing is that we simply grab the first solution that comes into our heads. This first solution lines up with some of the evidence, or at least the evidence that is easy to find. But it also contains a bad mix of beliefs, prejudices, and ignorance.

With this preconception (our explanation) firmly stuck in our minds, our search for further evidence is tainted. As the author says, we simply get off at the first bus stop no matter what. More supporting evidence drives us deeper into our preconceived conclusion. When evidence does not fit our explanation, we ignore it or explain it away. We desire emotional closure so much, we want to end our search for truth even at the cost of the truth.

Some points to ponder

Some of the greatest scientific discoveries and technological advancements occurred because some scientist decided to take contrary evidence at face value. For years, scientists explained away the early experiments at measuring the speed of light as it related to the earth’s movement through the universe. Then Einstein and a few others thought that maybe the experiment didn’t fail. It didn’t prove what the experimenters hoped, an accurate measure of earth’s absolute and universal speed and trajectory. But if the experiment is valid, it means the speed of light is a constant. And so the theory of relativity was born. (I heard that the theory was initially called the theory of invarriance, since the speed of light was a new absolute in science.)

The biggest help in producing valid hypothesis testing is a methodical, plodding, step-by-step procedure. As with other decision procedures, a methodical and structured procedure greatly minimizes our human tendencies that get in the way.

We’ll begin describing a proper method for hypothesis testing in the next installment.