The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…


Direct vs. Indirect Evidence of Student Learning

Written by: Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences and Disorders at Illinois State University

measure2Later this week, I have the opportunity to facilitate a workshop on peer mentorship in SoTL at a nearby university. I solicited questions from mentor faculty as part of my workshop planning process. In doing so, one of the most interesting questions I received was the following: In studying student learning, how can teacher/learner perceptions be considered a reliable data source?

This question gets at an important consideration in the planning of a SoTL project. What is my source of evidence? Will I use data from focus groups, surveys, student reflections, or something else? Will this evidence focus on student self-reports/perceptions of learning or will the evidence be more objective? The best guidance is that your evidence should match the purpose of your SoTL study. If you are seeking to understand students’ perspectives on a learning experience, then the evidence you collect should align with this. If, however, you are seeking to measure student learning, other forms of data may be more advantageous.

When SoTL-ists talk about their data, they can generally ascribe one of two labels to their evidence: direct or indirect. Direct evidence comes from objective sources such as classroom artifacts (e.g., exams/quizzes, projects/assignments), systematic observations (e.g., video/in-person observations, photographs), or student reflections that tell the story of their own attitudes or beliefs. Indirect evidence is sourced from more subjective sources – student reports of their own learning, teacher reflections of student learning (Vanderbilt, 2013). So, to return to the excellent question posed to me above, teacher/learner perceptions CAN be a reliable data source if the SoTL work in question seeks to understand how teachers/learners feel about their learning. That said, if a researcher is seeking to identify changes in student learning, perceptions alone are not a strong form of evidence to study (see this blog post from 2015 for an expanded discussion of this notion).

One of the best resources I’ve found to explain the difference in various evidence types in SoTL was published by Vanderbilt University’s Center for Teaching. This resource, Gathering Evidence: Making Student Learning Visible, outlines the difference between direct and indirect evidence clearly and cogently, providing examples and brief explanations to understand these concepts well. For my upcoming workshop, I adapted and converted the information shared on this resource (giving ample credit to Vanderbilt!) into a decision tree to share with the SoTL mentors I’ll be working with. As SoTL mentors, they will need to be well informed as to the pros and cons of direct and indirect evidence. I’m hopeful this visual will give us a good starting point for that discussion!

Direct vs indirect decision tree

As a plug for upcoming blogs, additional information is coming in October and November on methods to consider evidence in new and different ways…stay tuned! I am certain that most of the methods that will be covered will apply predominantly to analysis of direct evidence in the study of teaching and learning.

Blog Reference

Vanderbilt University Center for Teaching. (2013). Gathering evidence: Making student learning visible. Available at:




Leave a comment

Outside the Box Pedagogies Supported by Emerging Evidence

Written by Jen Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences & Disorders at Illinois State University

Screen Shot 2017-08-21 at 11.04.16 AM.pngIt’s back…syllabus construction season! I have spent the last several weeks considering various instructional approaches for a class I’m teaching for the first time. My class — Assessment Across the Lifespan — is a clinically-based course intended to round out ISU’s two-year speech-language pathology graduate program. I’ve been planning the semester carefully these last several weeks, focusing on important “take aways” for students. As I matched instructional approaches with various course topics, I struggled to find a pedagogy that would allow me to facilitate the development of high-caliber observation skills — a critical tool in any clinical toolbox — in my students. To figure this out, I started looking toward relevant evidence to see what types of strategies/pedagogies were being used effectively to teach observational skills.

While I learned of varied evidence-based approaches to students learn to be better observers, there was one that resonated strongly with me. Jasani and Saks (2013) studied the impact of using visual art to help enhance the observational skills of medical students at their institution. Their study used a pre/post-test design to evaluate student observations before and after a three-hour visual observation strategy module that focused on using art to sharpen visual observation skills. While the number of observations did not differ from pre- to post- measures, students perceived they developed stronger clinical skills (content understanding and clinical mindfulness) as a result of this activity. I’m thinking I may use this approach to help my students sharpen their observational skills this term…and perhaps evaluate the impact of this pedagogy on student learning in speech-language pathology.

As I was reading about the arts-based approach for clinical teaching, I came across a blog that detailed the use of one of my favorite TV shows, the Amazing Race, to teach cultural geography to students. Sarah Smiley reflected on the use of this approach in a recent issue of the Journal of Geography (2017). In great detail, Smiley explained her reasoning for selecting various shows (e.g., to teach about language or religion) and discussed how her course structure allowed for active in- and out-of-class learning experiences. She identified student learning barriers and work-arounds for subsequent applications of this pedagogy. Overall, while no learning data was provided, Smiley allowed for a very honest look into the development of and reflection on an “out of the box” pedagogy. A bit of digging turned up a similar type of course autopsy by Smiley and Post (2014) in which the use of popular music to engage students in the study of introductory geography is studied.

Thinking about one more, evidence-based, “out of the box” approach to teaching, I was reminded of the work my ISU colleagues Bill Anderson, Sarah Bradshaw, and Jennifer Banning (2017). They studied a “twist” on case-based learning that yielded interesting possibilities for course instructors focused on change or development over time. This pedagogy, the interrupted case study (ICS), allows for case studies to unfold over time, with course instructors releasing selected and organized parts of each case progressively to disclose important aspects of the case as a sort of problem-based learning experience over time (Anderson et al, 2017). For this investigation, researchers used a video case study in a human development course to follow a cohort of individuals through their lifetimes for a period of 50 years. Segments of the video case study were played over the course of the semester. In between video segments, students were tasked with applying, discussing, and comparing/contrasting relevant developmental theories germane to the videos they watched. Students were also asked to make predictions of what they might see during the next video segment that was released. Student reflections from across the semester were studied systematically to understand the impact of ICS. Preliminary findings indicated that the use of ICS has the potential to create the “need to know” in students, to connect theory to practice, and to raise students’ levels of critical thinking.

The instructional approaches discussed in this blog certainly are “outside the box” and have presented emerging evidence for their efficacy which provide a foundation for future inquiry to understand the comprehensive impacts of these pedagogies. I am appreciative of the work of innovators in teaching and learning such as those featured above. Their efforts often change my perspective and provide new ways of thinking about my teaching and my students’ learning, which is always a good thing, particularly at the start of a new semester! Happy fall term to all!

Blog References

Anderson, J. W., Bradshaw, S., & Banning, J. (2017). Using interrupted video case studies to teach developmental theory: A pilot study. Gauisus, 4.

Jasani, S. K. & Saks, N. S. (2013). Using visual art to enhance the clinical observation skills of medical students. Medical Teacher, 35(7).

Smiley, S. L. (2017). Teaching cultural geography with the Amazing Race. Journal of Geography, 116(3), 109-118.

Smiley, S. L. & Post, C. W. (2014). Using popular music to teach the geography of the United States and Canada. Journal of Geography, 113(6), 238-246.

Leave a comment

Class Discussions – Ideas for Use and Study

Written by Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences & Disorders at Illinois State University

At EuroSoTL in June, I had the opportunity to attend a workshop session titled “Construction as a tool for reflection – A LEGO workshop,” developed by Dr. Staffan Andersson and Dr. J. Andersson Chronholm of Uppsala University in Sweden. The workshop allowed attendees to use LEGO Serious Play in exploring and discussing issues related to SoTL. I can certainly say we did discuss issues that are important in the SoTL world. But in doing so, we had FUN meaningfully engaging with each other as we told our SoTL stories (stay tuned – Dr. Andersson has agreed to write a blog on the experience with photos of our LEGO work in an upcoming SoTL Advocate blog!). On my plane ride home from the conference, I pondered the LEGO workshop, wondering how I could similarly engage my students in thinking about important disciplinary issues in unexpected ways.

Discussion bookBack in my office, I recalled a book I had purchased recently by Stephen Brooksfield and Stephen Preskill titled The Discussion Book: 50 Great Ways to Get People Talking – and sat down to read. Different from more familiar teaching/learning handbooks and resources, this book focuses exclusively on engaging in discussions across a variety of contexts for a wide range of purposes – a topic with an appeal to both public and private sector stakeholders: managers, employees, volunteers, teachers, and students. Looking at the techniques explained in the book, I noted some overlap (e.g., Think-Pair-Share or Critical Debate) with popular teaching/learning books such as Collaborative Learning Techniques: A Handbook for College Faculty (Barkley, Cross, & Major, 2005); however, plenty of “new to me” ideas also were set forth by the authors.

In terms of organization, Brookfield and Preskill’s book includes ideas to accomplish the following:

  • Get discussion going with new groups
  • Promote good questioning
  • Foster active listening
  • Hold discussions without speaking
  • Get people out of their comfort zone
  • Engage in a text-based discussion
  • Democratize participation
  • Transition from small to large groups
  • Building group cohesion
  • Making group decisions

Each technique is presented in a formulaic manner within its own chapter in the book. Each chapter contains the following sections:

  • Purpose of technique
  • How it works
  • Where and when it works well
  • What users appreciate
  • What to watch out for
  • Questions that fit the technique

While I found several techniques that look promising for use with my classes and students this term (particularly a technique termed “single word sum-ups” to help my students speak briefly, concicely, and find themes across classmates), I was struck by the lack of any real evidence presented to accompany these. Great ideas? The book has many. Evidence to suggest that the techniques explained are effective? That was most definitely lacking.

What does that mean for us as SoTL enthusiasts? Well, thinking specifically of McKinney’s (2007) teaching continuum, this book could appeal to good teachers who apply these techniques with thought and care, scholarly teachers who seek evidence elsewhere to support the use of these strategies prior to applying them, and scholars of teaching and learning who take the opportunity to engage in classroom-based SoTL to systematically study the effectiveness of the techniques they choose to apply. From a teaching and learning standpoint, it is the case that this book offers potential benefit to many (students, teachers, scholars). So perhaps it truly does offer something for everyone. However, I DO hope that some instructors are tempted to study the outcomes of using any techniques they try! Such a study might be the perfect opportunity for a student or faculty member looking to engage in their first (or 50th!) SoTL experience.


Blog references:

Anderson, S. & Anderson Chronholm, J. (2017, June). Construction as a tool for reflection: A LEGO workshop. Presentation at EuroSoTL in Lund, Sweden.

Barkley, E. F., Cross, K. P., & Major, C. H. (2005). Collaborative learning techniques: A handbook for college faculty. San Francisco: Jossey-Bass.

Brookfield, S. D. & Preskill, S. (2016). The discussion book: 50 great ways to get people talking. San Francisco: Jossey-Bass.

McKinney, K. (2007). Enhancing learning through the scholarship of teaching and learning: The challenges and joys of juggling. San Francisco: Jossey-Bass.

Leave a comment

Call for University-Wide SoTL Award Open

Applications are sought for the 2018 Dr. John Chizamr & Dr. Anthony Ostrosky Scholarship of Teaching and Learning Award. This award recognizes faculty and academic staff at ISU who have contributed to the field of SoTL, the SoTL body of knowledge, improved teaching, and enhanced learning.

Applications should be submitted by Monday, November 13, 2017. Requirements for application are detailed below. Information about past award recipients and application procedures can be found on the Cross Chair website, as well. Please contact Jen Friberg ( with questions about this award.

SoTL Award18