By Kathleen McKinney, Cross Endowed Chair in SoTL, Illinois State University
Often when I am facilitating workshops on designing, conducting, and sharing scholarship of teaching and learning empirical projects, I am questioned about the methodological rigor of SoTL research. Such questions are usually from participants in disciplines that emphasize quantitative research and/or experimental methodologies and/or large sample sizes and/or the use of known and tested instruments in their disciplinary research. Sometimes they are traditional educational researchers using such techniques often in K-12 settings. They are concerned, understandably, with reliability, internal validity, and external validity of SoTL studies.
Of course, there is SoTL research that uses quantitative research or experiments or instruments with known statistical characteristics or large samples. These SoTL studies, however, are atypical rather than the norm. And those of us who do SoTL are concerned with rigor, and with reliability and validity. But, we may think about these issues somewhat differently or emphasize other aspects of our projects over the traditional view of these methodological standards. I also believe, however, that there is overlap in how SoTL researchers and other researchers attempt to enhance reliability and validity.
Usually, reliability is defined as consistency, that is, the extent to which a measurement instrument or other tool yields essentially the same result when used more than once. Of course, this assumes no real change has taken place in what is being measured. Thus, as a hypothetical example, if you have an instrument to measure a student’s ability to engage in metacognition and they score in the ‘medium-high range’, when you give that student the same measure two weeks later, they should still score in the ‘medium-high range’ (assuming nothing happened that actually significantly decreased or increased this ability).
Internal validity is the degree to which a measure actually measures what it is supposed to or claims to measure. Thus, for example, if you have a measure to assess the quality of a student’s analysis of historical text, the measure should in fact be measuring the quality of that historical analysis and this should be demonstrated in a variety of ways (e.g., face, construct, criterion validity). The measure should not be measuring something else such as the student’s ability to write well or to tell a good story or to think critically (though those may correlate or overlap to some degree with your measure if they are related to or part of your construct of historical analysis). Internal validity also has a meaning in experimental designs in terms of the quality of the manipulations, measures, and design to give us clear results related to cause and effect; ruling out confounding factors and alternative explanations of the results.
External validity deals with the extent to which we can legitimately generalize the results of our study beyond that study. If our study was replicated or adapted, would the same results be found? We may want or be expected to have our results generalize to other contexts or situations, samples, measures of variables, etc. To what extent, for instance, does our finding that the use of reflective learning logs enhances student learning in our sociology senior capstone class generalize to other classes or students or disciplines or institutions?
There are two answers (at least) to the questions about SoTL lacking rigor in terms of reliability, and internal and external validity. The first answer is that SoTL does concern itself with these traditional methodological standards. We, similar to many other researchers in different fields, attempt to strengthen these qualities of our research by doing the following:
- building on prior research,
- using theory to guide our work,
- using established reliable and valid measures if they exist for our constructs or variables,
- making use of multiple measures of the same concept or behavior or outcome we are studying,
- making use of multiple methods (often both qualitative and quantitative) so as to triangulate our results,
- engaging and honoring student voices in our SoTL projects in terms of both their descriptive data and as members of our research teams,
- replicating others’ and our own SoTL research, and
- seeing a single SoTL study as only one piece of a large and complex puzzle in a long-term, multi-study SoTL research agenda.
The second answer to such questions, especially to the concern about external validity or generalizability, is to remind others that SoTL is a special type of research and scholarly activity. SoTL is applied, action, practitioner research. The purposes are to learn about or teaching and, more so, our students’ learning and other outcomes in our classes and programs. And to use what we learn to enhance the learning and development of those and future students. Because of these purposes, SoTL is context-specific in terms of the specific students or tasks, the learning we are attempting to measure, or the discipline, program or institution.
I argue, then, that SoTL research does not inherently lack rigor. It is not, by definition, ‘weaker’ than other research. It is just a little different!