The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…

Is SoTL ‘Less Rigorous’ or Simply ‘Different’ than Other Research?


By Kathleen McKinney, Cross Endowed Chair in SoTL, Illinois State University

Often when I am facilitating workshops on designing, conducting, and sharing scholarship of teaching and learning empirical projects, I am questioned about the methodological rigor of SoTL research. Such questions are usually from participants in disciplines that emphasize quantitative research and/or experimental methodologies and/or large sample sizes and/or the use of known and tested instruments in their disciplinary research. Sometimes they are traditional educational researchers using such techniques often in K-12 settings. They are concerned, understandably, with reliability, internal validity, and external validity of SoTL studies.

Of course, there is SoTL research that uses quantitative research or experiments or instruments with known statistical characteristics or large samples. These SoTL studies, however, are atypical rather than the norm. And those of us who do SoTL are concerned with rigor, and with reliability and validity. But, we may think about these issues somewhat differently or emphasize other aspects of our projects over the traditional view of these methodological standards. I also believe, however, that there is overlap in how SoTL researchers and other researchers attempt to enhance reliability and validity.

Usually, reliability is defined as consistency, that is, the extent to which a measurement instrument or other tool yields essentially the same result when used more than once. Of course, this assumes no real change has taken place in what is being measured. Thus, as a hypothetical example, if you have an instrument to measure a student’s ability to engage in metacognition and they score in the ‘medium-high range’, when you give that student the same measure two weeks later, they should still score in the ‘medium-high range’ (assuming nothing happened that actually significantly decreased or increased this ability).

Internal validity is the degree to which a measure actually measures what it is supposed to or claims to measure. Thus, for example, if you have a measure to assess the quality of a student’s analysis of historical text, the measure should in fact be measuring the quality of that historical analysis and this should be demonstrated in a variety of ways (e.g., face, construct, criterion validity). The measure should not be measuring something else such as the student’s ability to write well or to tell a good story or to think critically (though those may correlate or overlap to some degree with your measure if they are related to or part of your construct of historical analysis). Internal validity also has a meaning in experimental designs in terms of the quality of the manipulations, measures, and design to give us clear results related to cause and effect; ruling out confounding factors and alternative explanations of the results.

External validity deals with the extent to which we can legitimately generalize the results of our study beyond that study. If our study was replicated or adapted, would the same results be found? We may want or be expected to have our results generalize to other contexts or situations, samples, measures of variables, etc. To what extent, for instance, does our finding that the use of reflective learning logs enhances student learning in our sociology senior capstone class generalize to other classes or students or disciplines or institutions?

There are two answers (at least) to the questions about SoTL lacking rigor in terms of reliability, and internal and external validity. The first answer is that SoTL does concern itself with these traditional methodological standards. We, similar to many other researchers in different fields, attempt to strengthen these qualities of our research by doing the following:

  • building on prior research,
  • using theory to guide our work,
  • using established reliable and valid measures if they exist for our constructs or variables,
  • making use of multiple measures of the same concept or behavior or outcome we are studying,
  • making use of multiple methods (often both qualitative and quantitative) so as to triangulate our results,
  • engaging and honoring student voices in our SoTL projects in terms of both their descriptive data and as members of our research teams,
  • replicating others’ and our own SoTL research, and
  • seeing a single SoTL study as only one piece of a large and complex puzzle in a long-term, multi-study SoTL research agenda.

The second answer to such questions, especially to the concern about external validity or generalizability, is to remind others that SoTL is a special type of research and scholarly activity. SoTL is applied, action, practitioner research. The purposes are to learn about or teaching and, more so, our students’ learning and other outcomes in our classes and programs. And to use what we learn to enhance the learning and development of those and future students. Because of these purposes, SoTL is context-specific in terms of the specific students or tasks, the learning we are attempting to measure, or the discipline, program or institution.

I argue, then, that SoTL research does not inherently lack rigor. It is not, by definition, ‘weaker’ than other research. It is just a little different!






5 thoughts on “Is SoTL ‘Less Rigorous’ or Simply ‘Different’ than Other Research?

  1. Hi Kathleen! Thanks for this post. As you know, I am very interested in the question of ‘what is sotl’ and how do we explain it to our colleagues who are not yet engaged. It can be a struggle, true, but personally I don’t think labelling as SoTL ‘different’ or ‘special’ helps. There are plenty of research approaches in non-quantitative disciplines that we may borrow from in this work and where the debate about reliability/validity/rigor/trustworthiness/generalizability has long ago been addressed, including but not limited to, action research. In other words, SoTL is not different than ‘other research’, it is only (often) different from quantitative research. I think referring to these other research approaches, the disciplines they come from, and the kinds of questions about learning that they can be used to investigate might be helpful. We don’t have to reinvent the wheel!! Please see Miller-Young and Yeo (2015) for more. 🙂


    • Janice (and others), Hi. I don’t think we are really in disagreement. I have discussed for decades SoTL as quality research-similar to other research- using both quantitative and qualitative methods… Even in this post, I note that SoTL is like other research in many ways. Also, I was not concerned, in this post, with the understanding of SoTL by those in non-quantitative disciplines or those doing a form of action research; mostly, these researchers get SoTL methodologically already and don’t need help understanding how/why we deal with reliability and validity. My point of calling SoTL ‘different’ was to acknowledge that it may be different IF you contrast it to a very narrow view of research as quantitative (and especially quantitative) large scale or experimental research (a point you also make) as well as in contrast to the basic (not action or applied) research that many people do in their disciplines. I think it is useful to remind people that SoTL is context-specific, local, applied, possibly qualitative…and for many very traditional researchers especially in certain disciplines, this is ‘diffferent.’ Again the point was to explain why and how ‘different’ should not be equated with less rigorous or weaker or bad research. There is nothing wrong with being ‘different’ in some ways!


      • Yes. We are essentially in agreement. 🙂 But why continue to contrast SoTL with quantitative, large scale and/or experimental research instead of drawing upon the vast body of qualitative research that we can use to support our case that yes, SoTL is in fact research i.e. it adds knowledge and advances practice.


      • Because of the particular audience I am trying to influence. So I believe I need both arguments.

        Liked by 1 person

  2. Providing some articles on qualitative research written specifically for STEM folk might be helpful. I like this one:


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s