The right order of method sections

Method sections in academic (psychology) papers usually consist of the following sections: Participants, Design, Procedure, and Materials. They also tend to be presented in this order. But is this, generally speaking, the right order? I don’t think so.

I think the proper order of Method sections is:

1. Design
2. Procedure
3. Materials
4. Data Analysis
5. Participants

Two things are notable here. One, there’s a Data Analysis section. Two, the Participants section is all the way at the end. Here I anticipate that your reaction will be that this is crazy, because that’s not how we do things. But that’s not a good enough reason of course. We should be thinking about whether the order makes sense in terms of whether the content of each section logically follows from each other. For some sections you first need to know information from the other sections in order for your decisions to make sense. Putting the Participants all the way at the beginning doesn’t make sense, and the reason for that is the power analysis.

Now that power analyses are getting more popular, psychologists have to try and make them fit in their Method section. But rather than thinking about what actually goes into a power analysis, and how to present that information to the reader, they generally stick to the format they’re used to. Or perhaps it’s because they misunderstand how a power analysis works, thinking that you only have 1 power analysis per study, so you should present it together with the Design of the study. Since the Design and Participants are sometimes combined, I can see how this might be the case. That still doesn’t make sense, though, and to understand that, we need to understand power analyses.

What goes into a power analysis? A power analysis consists of setting a few parameters, such as the effect size, alpha, and beta. The alpha and beta parameters are pretty constant across different power analyses, but the effect size isn’t. The effect size depends on the exact analysis you want to power for. A t-test is usually done with a Cohen’s d in mind, while a correlation test is done with a correlation in mind. With more sophisticated analyses, such as repeated measures analyses, you need to set additional parameters (e.g., the correlation between repeated measures). This means that your power analysis is dependent on the exact analysis you will do. Actually, a power analysis is always about a specific analysis, so by that logic alone, you should first present which analysis you will do. Not only that, but you also need to power for all analyses you do, not just 1. In other words, a power analysis is something that is tied to a statistical test, and not to the design of a study (which would mean you only need 1 power analysis per study). The result is obvious: you first need to discuss the analyses you want to run before you can talk about power. This means you need a Data Analysis section in your Method section. Here you can elaborate on the analyses you will run, which analyses are the primary ones that you want to power for, and perhaps elaborate on some secondary or exploratory analyses that you won’t power for. You can also use this section to then present the power analysis. After that you get your needed sample size and you can start to explain how you obtained that sample size (i.e., the Participants section).

What do you need to know in order to understand the Data Analysis section? That would be the Design and Materials. You need to know about the design to know whether it is, for example, a between-subjects design or a within-subjects design. You also need to know what the independent variables and dependent variables are. More specifically, you want to know how they are measured. How many levels are there in the independent variables? Is the outcome measure categorical or continuous? These are some of the properties of the measures that determine the appropriate analysis technique. This, in turn, means the Design section and the Materials section need to come before the Data Analysis section.

Putting all of this together, I think it makes the most sense to begin with the Design, followed by the Procedure and Materials (possibly combined). This should be followed by a Data Analysis section that includes the analyses and associated power analysis (for all primary analyses, at least). Once these aspects are known, it makes sense to end, rather than start, with the Participants section. So the right order of Method sections is:

1. Design
2. Procedure
3. Materials
4. Data Analysis
5. Participants