a. Your project was already designed and planned before you joined the lab

In this case, all there is left for you is to learn about the project. If available, ask to have a copy of the proposal of the project, as that will include all details you need to know.

If a formal proposal is not available/accessible, then ask your advisor to provide you with some information like:

• What is the knowledge gap that your research is trying to fill?

• What are the hypotheses behind your project?

• What are the objectives of the project?

• What is the treatment design (the part of the design that addresses the study hypotheses) of your study?

• What is the timeline of your project?

• What are the expected deliverables of your project?

b. You have a clean slate to design your own project

In this case, to a certain extent, you have the freedom to propose a study and plan it from scratch. How nice!

The timeline here should be something like the following:

• Identify the knowledge gap based on the literature.

• Define clear hypotheses about your topic.

• Define objectives based on your hypotheses.

• Think about what treatments you should select to properly answer your hypotheses. The answer to this question will make up the treatment design of your study.

• Think about what limitations on experimental material you may have. The answer to this question will determine the experimental design of your study.

For a more in-depth discussion on the differences between treatment and experimental design, check this post.

a. How do we use statistical design and analysis to further understand the story of our data?

Proper statistical design and analysis is critical in understanding your data in the most unbiased way possible.

That is because your treatment design will help you to directly answer your objectives and hypotheses (assuming they were selected properly), while your experimental design will control for experimental material heterogeneity previously identified and incorporated in it (that otherwise would be noise in your data).

Therefore, it is critical that you understand how to translate what happened on the field (related to design) to statistical programming language.

Always remember that the statistical software, whichever you may be using, does not know what design you used, and will ALWAYS give an answer, regardless of how different the model is from what you designed on the field.

Thus, it is up to you (the researcher) to properly translate field design to statistical programming language.

Wrongly specified models can give you erroneous results, especially standard errors, ANOVA p-values, and pairwise comparison p-values!

b. Can you go ‘too far’ in statistical analysis when trying to understand data/story from the data?

Yes. That has been referred in the literature as “p-hacking” or “researcher degrees of freedom”. For a thorough read on this, I refer your to

Gelman, A. & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time.

which can be read here.

In a nutshell, we should strive to define protocols for data cleaning/processing, statistical analyses types, model specifications (based on your study design), alpha values, adjustment choices, and anything else data-analysis-related AHEAD OF TIME.

If you end up changing plans as you progress on your analysis, ask yourself “Is this new choice simply to make my results significant?”. Hopefully you will have a great reason that will not simply be to make your results “better” whatever that may mean.

Also, defining clear hypotheses and objectives early in the process will help that you start off in the right direction, instead of “shooting” into multiple directions that could end up answering a different objective.

c. How to interpret your analysis?

Analyses results should always be interpreted based on a statistical test, and better yet if accompanied by some sort of variability measure.

Generally, model means (in case of ANOVAs) and coefficients (in case of regressions) are presented alone, followed only by a significance test result (like a letter or asterisk indicating significance at certain alpha value).

While this is ok, your results will be much better presented if you include standard errors, confidence interval, etc.

Also, plots that include both analysis results (e.g., mean, standard error, letter) AND raw data are the best option for communicating your data clearly. Plot types that can be used here include boxplots, violin plots, density plots, bee-swarm plots, and others.

For some more resources on this, just google “ggplot density plot” (as an example) and you will have a plethora of cases to use.