The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…


1 Comment

Study Design and Data Analysis in SoTL

Written and compiled by Jennifer Friberg, Cross Endowed Chair in SoTL and Professor of Communication Sciences & Disorders at Illinois State University

In June, I visited the University of South Alabama (USA) and worked with Raj Chaudhury and Sue Mattson to get a group of faculty started with their year-long SoTL Academy efforts. Approximately 30 faculty from across USA’s campus came together to learn about SoTL and plan a SoTL project. We spent two days together in workshops and consultations and all participants left with a draft plan for SoTL work they hoped to conduct this current academic year.

This was the second year I was able to join the USA crew for this two-day educational and research development event. Sue and I agreed that a resource that would be valuable for the USA faculty for the second iteration of the SoTL Academy would be something that helped social science-oriented researchers see how SoTL might dovetail with concepts and ideas they already understood well. Thus, the following grids focused on descriptive, correlational, and experimental/quasi-experimental design were drafted and used in discussions about how SoTL might look like participants’ own disciplinary research — and how it might not. This resource is being shared here now, in the hopes that others might find this information valuable, as well.

Should you wish to obtain a copy of this information in PDF form, please feel free to email me at jfribe@ilstu.edu. I’m happy to share!

Descriptive Research
Description of Study Design Descriptive research characterizes a group of people, a context, or a phenomenon. These studies do not seek to establish a causal relationship; rather, they provide information about “what is” occurring or being observed regarding the focus of study.

Descriptive studies include observational, case study, and survey methods.

Exemplar SoTL Projects
  • Survey students’ re: practices in using print vs. online textbooks to support learning.
  • Observe how students’ use of technology in the classroom impacts attention span.
  • Study high achieving students in a course to predict practices/variables of success to share with future students.
Qualitative Analysis Options Qualitative data in a descriptive study is reported as narrative, reflection, open-ended response, field note, etc. Such data will need to be further analyzed for themes, categories, or patterns.

Common qualitative approaches in descriptive SoTL research include: case studies, action research processes, analytic induction, ethnography, comparative analysis, frame analysis, grounded theory, and interpretive phenomenology, among others.

Quantitative Analysis Options Quantitative data in a descriptive study is often reported in the form of descriptive statistics (e.g., mean, median, mode) along with standard deviations. Statistics might be used here, depending on the data collected and the topic being studied.

These data might emerge from test scores, grades on a course assignment or project, survey data, or frequency data.

 

Correlational Research
Description of Study Design Correlational research seeks to determine whether a relationship exists between two or more variables, but cannot determine if one variable causes another. Variables aren’t manipulated; rather, they are observed to determine any relationship that might exist between them.

Note that some sources identify correlational research as a quantitative-only subset of descriptive research, as some descriptive research might suggest a correlation found via grounded theory or other qualitative methods of research.

Exemplar SoTL Projects
  • Determine the relationship between number of hours studying and success on a quiz/test.
  • Identify whether there is a link between the use of peer editing and performance on a writing assignment.
  • Understand whether the use of social media helps students to summarize course content effectively.
Qualitative Analysis Options Qualitative data analysis is not undertaken for correlational research, as numerical data is needed to calculate a correlation coefficient.
Quantitative Analysis Options Correlational research is a quantitative method of inquiry. Correlation can only be determined for quantifiable data. These are data in which numbers are meaningful, usually quantities of some sort. It cannot be used for purely categorical data, such as gender, brands purchased, or favorite color.

Statistics are used to determine a correlation coefficient to identify positive, negative, or zero correlation. One thing to keep in mind is that any identified correlation does not mean that one variable caused the other to react. Instead, correlations simply define that a relationship exists.

 

Experimental/Quasi-Experimental Research
Description of Study Design Experimental and quasi-experimental research designs seek to manipulate one variable and control all others to investigate cause/effect relationships. All participants are assigned to either a control or experimental group. An intervention is applied to the experimental group. The control group has no intervention applied.

The key difference between experimental and quasi-experimental designs is the concept of randomization. If participants are assigned to control and experimental groups randomly, the research design is experimental. Non-random group assignment yields a quasi-experimental research design. True experimental research is considered the gold standard of research by many researchers, because random group assignment leads to optimal internal validity. In situations where random group assignment is not possible or ethical, quasi-experimental designs offer an alternative that allows the research to continue and still produce valid results.

Almost no SoTL qualifies as truly experimental in nature due to inherent ethical and logistical characteristics of SoTL that makes this type of research difficult to conduct (e.g., true randomization). One of the most common quasi-experimental designs for SoTL research is the pre-test/post-test with no control group design.

Exemplar SoTL Projects
  • Does the use of simulated patients help nursing students improve observational skills?
  • Do architecture students who initially design structures by hand understand the concept of space more deeply?
  • Do history students exposed to guided reading demonstrate a deeper understanding of historical imagination?
Qualitative Analysis Options Experimental and quasi-experimental designs may yield data that is descriptive (e.g., surveys, interviews, observations) that require qualitative analyses. Similar to information provided above for descriptive research, any qualitative data will need to be further analyzed for themes, categories, or patterns.

Common qualitative approaches to data analysis in SoTL include: case study, action research processes, analytic induction, ethnography, comparative analysis, frame analysis, grounded theory, and interpretive phenomenology, among others.

Quantitative Analysis Options Experimental design lends itself to more straightforward and simpler types of statistical analysis. Primarily due to the lack of randomization, quasi-experimental studies usually require more advanced statistical procedures. Quasi-experimental designs may also utilize surveys, interviews, and observations which may further complicate the data analysis.

Quantitative analysis requires several steps. First numeric data is assigned a level (nominal, ordinal, interval, or ratio). Next, descriptive statistics are calculated for data (e.g., means, standard deviations). For some studies, descriptive statistics may be adequate; however, if you want to make inferences or predictions about your population, inferential statistics (e.g., t-test, ANOVA, regression) may be indicated.

Blog References:

Bishop-Clark, C. & Dietz-Uhler, B. (2012). Engaging in the scholarship of teaching and learning: A guide to the process and how to develop a project from start to finish.        Stylus: Sterling, VA.

Campbell, D. T. & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Cengage: Boston.

Cresswell, J. W. (2012). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Sage: Thousand Oaks, CA.

Gurung, R. A. R. & Wilson, J. H. (Eds.). (2014). Doing the scholarship of teaching and learning: Measuring systematic changes to teaching and improvements in learning. Jossey-Bass: San Francisco.

 

 

Advertisements


Leave a comment

Decoding was a Success!

Written by Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences & Disorders at Illinois State University

Late last week, a total of 41 faculty from ISU participated in one of two Decoding the Disciplines events on campus. Sponsored by the Office of the Cross Endowed Chair in SoTL, these events featured Dr. David Pace, Emeritus Professor of History at Indiana University and co-creator of an approach to spanning the novice-to-expert gap called “Decoding the Disciplines.”

First, an event for faculty in ISU’s Department of History was held at Milner Library. Nineteen faculty joined in a discussion about SoTL and Decoding the Disciplines. They worked to identify bottlenecks in their curriculum where a Decoding approach might be beneficial to supporting student learning and curriculum planning. Attendees were privy to the first-ever whole group Decoding interview, where Dr. Pace simultaneously interviewed the entire faculty to identify whole program bottlenecks for future attention and focus.

Screen Shot 2018-04-09 at 11.52.29 AM.png

History faculty engrossed in small group discussions about disciplinary bottlenecks

The following day, 22 faculty from across campus experienced a full-day Decoding workshop, learning about each of the seven steps of the process. Participants identified student learning bottlenecks one or more of their classes, then brainstormed together on approaches for Decoding interviews and possibilities for collecting and sharing data to reflect pre- versus post-Decoding student learning.

Screen Shot 2018-04-09 at 11.53.49 AM

ISU faculty learning about the steps of the Decoding the Disciplines process

The establishment of a Teaching/Learning Community to continue these Decoding conversations is underway. Specifically, faculty have expressed an interest in looking more deeply into:

  • The impact of bias in the identification of bottlenecks
  • The relationship between knowing and doing in courses where the essence of the experience is understanding process
  • Differences between faculty and student visions of a goal for a class, project, or assignment
  • Understanding ways to approach emotional bottlenecks

These Decoding experiences would not have been possible without the assistance and support received from the Office of the Provost, Ross Kennedy (Chair, Department of History at ISU), Richard Hughes (Associate Professor, History at ISU and co-planner of the History Department event), and Beth Welch.

A list of Decoding the Disciplines resources can be found in this recent blog post.


Leave a comment

Direct vs. Indirect Evidence of Student Learning

Written by: Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences and Disorders at Illinois State University

measure2Later this week, I have the opportunity to facilitate a workshop on peer mentorship in SoTL at a nearby university. I solicited questions from mentor faculty as part of my workshop planning process. In doing so, one of the most interesting questions I received was the following: In studying student learning, how can teacher/learner perceptions be considered a reliable data source?

This question gets at an important consideration in the planning of a SoTL project. What is my source of evidence? Will I use data from focus groups, surveys, student reflections, or something else? Will this evidence focus on student self-reports/perceptions of learning or will the evidence be more objective? The best guidance is that your evidence should match the purpose of your SoTL study. If you are seeking to understand students’ perspectives on a learning experience, then the evidence you collect should align with this. If, however, you are seeking to measure student learning, other forms of data may be more advantageous.

When SoTL-ists talk about their data, they can generally ascribe one of two labels to their evidence: direct or indirect. Direct evidence comes from objective sources such as classroom artifacts (e.g., exams/quizzes, projects/assignments), systematic observations (e.g., video/in-person observations, photographs), or student reflections that tell the story of their own attitudes or beliefs. Indirect evidence is sourced from more subjective sources – student reports of their own learning, teacher reflections of student learning (Vanderbilt, 2013). So, to return to the excellent question posed to me above, teacher/learner perceptions CAN be a reliable data source if the SoTL work in question seeks to understand how teachers/learners feel about their learning. That said, if a researcher is seeking to identify changes in student learning, perceptions alone are not a strong form of evidence to study (see this blog post from 2015 for an expanded discussion of this notion).

One of the best resources I’ve found to explain the difference in various evidence types in SoTL was published by Vanderbilt University’s Center for Teaching. This resource, Gathering Evidence: Making Student Learning Visible, outlines the difference between direct and indirect evidence clearly and cogently, providing examples and brief explanations to understand these concepts well. For my upcoming workshop, I adapted and converted the information shared on this resource (giving ample credit to Vanderbilt!) into a decision tree to share with the SoTL mentors I’ll be working with. As SoTL mentors, they will need to be well informed as to the pros and cons of direct and indirect evidence. I’m hopeful this visual will give us a good starting point for that discussion!

Direct vs indirect decision tree

As a plug for upcoming blogs, additional information is coming in October and November on methods to consider evidence in new and different ways…stay tuned! I am certain that most of the methods that will be covered will apply predominantly to analysis of direct evidence in the study of teaching and learning.

Blog Reference

Vanderbilt University Center for Teaching. (2013). Gathering evidence: Making student learning visible. Available at: https://my.vanderbilt.edu/sotl/files/2013/09/4SoTLEvidence.pdf

 

 

 


Leave a comment

SoTL Book Information: ISU Faculty Authors

The announcement below is a press release from the Mathematical Association of America (MAA) that has just published a book on SoTL in Mathematics in higher education. This book could serve as a model for such books in other disciplines. In addition, chapter 17 was written by ISU faculty members in our department of Mathematics: Chapter 17. Mathematics Research Experiences for Preservice Teachers: Investigating the Impact on Their Beliefs by Wendy A. O¹Hanlon, David D. Barker, Cynthia W. Langrall, John A.Dossey, Sharon M. McCrone, and Saad I. El-Zanati. Another book focusing on SoTL in a particular discipline is also a model and one author is also an ISU faculty member (Jennifer Friberg): Ginsberg, S., Friberg, J., & Visconti, C. 2012. Scholarship of Teaching and learning in speech-language and audiology: Evidence-based education. San Diego: Plural Publishing. It is great to see ISU faculty take leadership roles in promoting SoTL in their disciplines.

A new book from the Mathematical Association of America (MAA) serves as a how-to guide for collegiate mathematics faculty who want to know more about conducting scholarly investigations into their teaching and their students’ learning. Out this month as part of the MAA’s Notes series, Doing the Scholarship of Teaching and Learning in Mathematics aims to both assist mathematics faculty interested in undertaking scholarly study of their teaching practice and promote a greater understanding of this work and its value to the mathematics community. The volume was envisioned and edited by Jacqueline Dewar and Curtis Bennett (Loyola Marymount University).

The Scholarship of Teaching and Learning (SoTL) movement encourages faculty to view difficulties encountered in the classroom as invitations to conduct research. In this growing field of inquiry, faculty bring their disciplinary knowledge and teaching experience to bear on questions of teaching and learning. They systematically gather evidence to develop and support their conclusions. The results are peer reviewed and made public.
The four chapters in Part I of Doing the Scholarship of Teaching and Learning in Mathematics provide background on this form of scholarship and specific instructions for undertaking a SoTL investigation in mathematics. Part II contains 15 examples of SoTL projects in mathematics from 14 different institutions, both public and private, spanning the spectrum of higher educational institutions from community colleges to research universities. The final chapter offers the editors’ synthesis of the contributing authors’ perceptions of the value of SoTL.

“Dewar and Bennett’s volume gives a vivid overview of the fresh field of the Scholarship of Teaching and Learning,” says Frank Farris (Santa Clara University). “It provides exactly what you need to get started doing research with your own classroom as the laboratory.”
(
Dewar, J., & Bennett, C. (Eds.). 2015. Doing the scholarship of teaching and learning in mathematics. Washington, DC: Mathematical Association of America.)