How to Evaluate Statistical Significance

Statistical hypothesis testing is guided by statistical analysis. Statistical confidence is calculated using a p-value that indicates the likelihood of an observation result when a certain proposition (the null hypothesis) is true. ^{[1] X Research Source} If the p-value is less than the significance level (usually 0.05), the experimenter can conclude that there is enough evidence to reject the null hypothesis and accept the conjecture. Using a simple t-test, you can calculate the p-value and determine the significance between two different groups of data.

Table of Contents

Set up your experiment

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/4/45/Assess-Statistical-Significance-Step-1-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-1-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/4/45/Assess-Statistical-Significance-Step-1-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-1-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Define your hypothesis. The first step in assessing statistical significance is to identify the question to answer and state the hypothesis. A hypothesis is a statement about the empirical data and possible differences in the population. Every experiment has a null and an inverse hypothesis. ^{[2] X Research Source} Generally speaking, you will compare two groups to see if they are similar or different.

Overall, the null hypothesis (H ₀ ) asserts that there is no difference between the two data groups. Example: Students who read the material before class did not score better at the end of the course.
The conjecture ( _Ha ) is the opposite of the null hypothesis and is the statement you are trying to back up with empirical data. Example: Students who read the material before class actually score better at the end of the course.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/1/17/Assess-Statistical-Significance-Step-2-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-2-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/1/17/Assess-Statistical-Significance-Step-2-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-2-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Select a significance level to determine how significant a difference can be in the data. The significance level (also known as alpha) is the threshold that you choose to decide on significance. If the p-value is less than or equal to the given level of significance, the data is considered to be statistically significant. ^{[3] X Research Sources}

As a rule of thumb, the significance level (or alpha) is usually chosen at the 0.05 level – meaning the chance of observing a difference seen on the data as random is only 5%.
The higher the confidence level (and therefore, the lower the p-value), the more significant the results.
If a more reliable data is required, lower the p-value to 0.01. Low p-values are often used in manufacturing to detect product defects. High reliability is important to accept that every part will function as it was designed to.
For most hypothesis-based experiments, a significance level of 0.05 is acceptable.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/7/73/Assess-Statistical-Significance-Step-3-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-3-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/7/73/Assess-Statistical-Significance-Step-3-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-3-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Decide whether to use a one-way test or a two-way test. One of the assumptions of the t-test is that your data is normally distributed. A normal distribution will form a bell curve with the majority of observations in the middle. ^{[4] X Research Sources} The t-test is a mathematical test conducted to check if your data lies outside of the normal distribution, above or below, in the “early” part of the curve.

If unsure whether the data is above or below the control group, use the two-way test. It allows you to check the significance level in both directions.
If you know what is the expected direction of the data, use the one-way test. In the example above, you expect the student’s grades to improve. Therefore, you use the one-way test.

Calculate standard deviation

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/c/ca/Assess-Statistical-Significance-Step-5-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-5-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/c/ca/Assess-Statistical-Significance-Step-5-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-5-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Determine the standard deviation formula. The standard deviation measures the dispersion of the data. It gives you information about the homogeneity of each data point in the sample. As a beginner, the equation can look quite complicated. However, the steps below will make it easier for you to understand the calculation process. The formula is s = √∑((x _i – µ) ² /(N – 1)).

s is the standard deviation.
∑ denotes that you will have to add up all the observations collected.
x _i represents each of your data values.
µ is the mean of the data for each group.
N is the total number of observations.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/3/31/Assess-Statistical-Significance-Step-6-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-6-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/3/31/Assess-Statistical-Significance-Step-6-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-6-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Average observations for each group. To calculate the standard deviation, you first need to calculate the mean of the observations in each individual group. This value is denoted by the Greek letter mu or µ. To do that, you simply add the observations together and divide by the total number of observations. ^{[7] X Research Sources}

For example, to find the average score of the pre-class reading group, let’s look at some data. To simplify, we will use a dataset of 5 points: 90, 91, 85, 83 and 94 (on a 100-point scale).
Add up all the observations: 90 + 91 + 85 + 83 + 94 = 443.
Divide the above sum by the number of observations N (N=5): 443/5 = 88.6.
The average score for this group was 88.6.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/d/d0/Assess-Statistical-Significance-Step-7-Version-3.jpg/v4-460px-Assess-Statistical-Significance- Step-7-Version-3.jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/d/d0/Assess-Statistical-Significance-Step-7-Version-3.jpg/ v4-728px-Assess-Statistical-Significance-Step-7-Version-3.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:” <div class=”mw-parser-output”></div>”}

Subtract the mean from each observed value. The next step involves the (x _i – µ) part of the equation. You will subtract the average value from each observation value. With the above example, we have five subtractions.

(90 – 88.6), (91 – 88.6), (85 – 88.6), (83 – 88.6) and (94 – 88.6).
The calculated value is 1.4; 2.4; -3.6; -5.6 and 5.4.

Determine statistical significance

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/d/d3/Assess-Statistical-Significance-Step-11.jpg/v4-460px-Assess-Statistical-Significance-Step-11. jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/d/d3/Assess-Statistical-Significance-Step-11.jpg/v4-728px-Assess-Statistical-Significance-Step- 11.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:”<div class=”mw-parser-output”></div> “}

Calculate the variance between your two groups of observations. Up to this point, the example has only dealt with a set of observations. To compare two groups, you obviously need data from both. Calculate the standard deviation of the second group of observations and use it to calculate the variance between the two experimental groups. The formula for calculating variance is: s _d = √((s ₁ /N ₁ ) + (s ₂ /N ₂ )). ^{[9] X Research Source}

s _d is the variance between groups.
s ₁ is the standard deviation of group 1 and N ₁ is the size of group 1.
s ₂ is the standard deviation of group 2 and N ₂ is the size of group 2.
In our example, suppose the data from group 2 (students who do not read before class) has a size of 5 and a standard deviation of 5.81. The variance is:
- s _d = ((s ₁ ) ² /N ₁ ) + ((s ₂ ) ² /N ₂ ))
- s _d = √(((4.51) ^2/5 ) + ((5.81) 2/5)) = √((20.34/ ⁵ ) + (33.76/5)) = √(4.07 + 6.75) = √10.82 = 3.29 .

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/1/14/Assess-Statistical-Significance-Step-12.jpg/v4-460px-Assess-Statistical-Significance-Step-12. jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/1/14/Assess-Statistical-Significance-Step-12.jpg/v4-728px-Assess-Statistical-Significance-Step- 12.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:”<div class=”mw-parser-output”></div> “}

Calculate the t-statistic of the data. The t-statistic allows you to convert data into a form that can be compared with other data. The t-value also allows you to perform a t-test, a test that allows you to calculate the likelihood that two groups are statistically different from each other. The formula for calculating the t-statistic is: t = (µ ₁ – µ ₂ )/s _d . ^{[10] X Research Source}

_µ1 is the mean of the first group.
_µ2 is the mean of the second group.
s _d is the variance between observations.
Use the larger mean as µ ₁ so as not to get a negative t-statistic.
For our example, let’s say the average observation of group 2 (those who didn’t read the previous post) is 80. The t-statistic is: t = (µ ₁ – µ ₂ )/s _d = (88, 6 – 80)/3.29 = 2.61.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/3/3b/Assess-Statistical-Significance-Step-13.jpg/v4-460px-Assess-Statistical-Significance-Step-13. jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/3/3b/Assess-Statistical-Significance-Step-13.jpg/v4-728px-Assess-Statistical-Significance-Step- 13.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:”<div class=”mw-parser-output”></div> “}

Determine the degrees of freedom of the sample. When using the t-statistic, the degrees of freedom are determined based on the sample size. Add the number of observations for each group and then subtract two. For the example above, the degrees of freedom (df) is 8 because there are 5 observations in the first group and 5 observations in the second group ((5 + 5) – 2 = 8). ^{[11] X Research Source}

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/c/c6/Assess-Statistical-Significance-Step-14.jpg/v4-460px-Assess-Statistical-Significance-Step-14. jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/c/c6/Assess-Statistical-Significance-Step-14.jpg/v4-728px-Assess-Statistical-Significance-Step- 14.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:”<div class=”mw-parser-output”></div> “}

Use the t-table to evaluate the significance level. Table of t-statistics ^{[12] X Research sources} and degrees of freedom can be found in standard statistics books or online. Find the line containing the degrees of freedom of the data and the p-value corresponding to the t-statistic you have.

With 8 degrees of freedom and t = 2.61, the p-value for the one-way test is between 0.01 and 0.025. Because the chosen significance level is less than or equal to 0.05, our data is statistically significant. With this data, we reject the null hypothesis and accept the inverse hypothesis: ^{[13] X Research source} students who read the material before coming to class had higher final grades.

{“smallUrl”:”https://www.wikihow.com/images_en/thumb/1/1b/Assess-Statistical-Significance-Step-15.jpg/v4-460px-Assess-Statistical-Significance-Step-15. jpg”,”bigUrl”:”https://www.wikihow.com/images/thumb/1/1b/Assess-Statistical-Significance-Step-15.jpg/v4-728px-Assess-Statistical-Significance-Step- 15.jpg”,”smallWidth”:460,”smallHeight”:345,”bigWidth”:728,”bigHeight”:546,”licensing”:”<div class=”mw-parser-output”></div> “}

Consider conducting further research. Many researchers do a baseline study with a few metrics to understand how to design a larger study. Doing other research with more measurable values will increase confidence in your conclusions.

Advice

Statistics is a large and complex field. Take a statistical hypothesis testing course equivalent to high school or college level (or higher) to understand statistical significance.

Warning

This analysis focuses on the t-test to test the difference between two normally distributed populations. Depending on the complexity of the data, you may need a different statistical test.

Steps

Set up your experiment

Calculate standard deviation

Determine statistical significance

Advice

Warning

Related Posts