The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…

Sources, Types, and Analysis of Data in SoTL

Leave a comment

Written and compiled by Jennifer Friberg, Cross Endowed Chair in SoTL, and Professor of Communication Sciences & Disorders at Illinois State University

A few weeks ago, I published a blog titled “Study Design and Data Analysis in SoTL,” which provided a resource for viewing different types of research designs and their application for the scholarship of teaching and learning (SoTL). This resource was developed as a part of preparations for a two-day “intro to SoTL” workshop at the University of South Alabama. Today’s blog shares a related resource created for the same workshop series.

dataWhile research design is a really important consideration in the planning of SoTL projects, I would argue that an equally important consideration is the determination of the type(s) of data that could be collected to address an identified research purpose/question, as different types of data can provide different types of narratives to describe teaching and learning. The tables below explore eight different data types commonly utilized by individuals completing SoTL studies. Each describes a data type, talks about the data yielded from each type, indicates whether qualitative and/or quantitative analysis is possible for the data source in question, and provides extra information to consider possible to using any of these data sources. Please note that this is not an exhaustive list of possible data sources for SoTL research!

Should you wish to obtain a copy of this information in PDF form, please feel free to email me at jfribe@ilstu.edu. I’m happy to share!

Survey Data

Description Surveys collect data to reflect participant perceptions or knowledge about a particular phenomenon at one point in time.
Data Potentially Yielded Data generated via a survey are answers to specific questions drafted and administered. Survey questions can be closed ended (e.g., multiple choice answers or Likert-type scale data) or can be open ended in nature. Different question types yield different data.
Quantitative Analysis Possible? Yes
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • Look for previously validated surveys to use as part of your study to increase validity/reliability of data collected.
  • For non-validated surveys, consider soliciting 2-3 expert reviewers to provide feedback re: survey content and format.
  • If survey is collecting indirect data (e.g., student perceptions), consider a plan to triangulate these data with a different, more directly objective source of information.

Interviews/Focus Groups

Description Considered a subset of survey research, these methods gather information about participant knowledge and feelings individually or with a group of people in a manner that allows (in some designs) for follow-up questions and non-standard data collection.  Interviews are generally conducted with a single person, while focus groups are group interviews.
Data Potentially Yielded Interactions occurring within interviews and focus groups are typically audio or video recorded. Orthographic transcriptions of these interactions can be created and analyzed to identify relevant trends across participants. Observations of specific behaviors might be quantified, as well, depending on the intent of the study’s design.
Quantitative Analysis Possible? Yes, but less frequent than qualitative analysis
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type Collection of interview and focus group data often causes extra human subjects review board scrutiny due to threats to confidentiality and/or anonymity. Consider how you will protect and explain protections for your study participants as part of your IRB development process.

Think Alouds

Description Think alouds are specific types of interviews where participants are asked to verbalize thoughts for internal cognitive processes in a sequential manner (e.g., how to complete a professionally-oriented task)
Data Potentially Yielded Like with interviews/focus groups, think alouds are typically audio or video recorded so that orthographic transcriptions of these interactions can be created and analyzed to identify relevant trends across participants.
Quantitative Analysis Possible? Yes, but less frequent than qualitative analysis
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type Similar to interviews and focus groups, collection of think aloud data often causes extra human subjects review board scrutiny due to threats to confidentiality and/or anonymity. Consider how you will protect and explain protections for your study participants.

Pre-/Post-Tests

Description Pre/post tests allow for collection of data to reflect changes resulting from some sort of intervention or experience over a pre-determined span of time.
Data Potentially Yielded Data collected is intended to reflect any changes (either positive or negative) resulting from an intervention or experience. Pre-/post-test data could be collected via a survey, reflection, or interview/focus group. The key here is that there are two (or more) sets of data to reflect differences across time.
Quantitative Analysis Possible? Yes
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • Look for previously validated instruments to use for pre/post test designs.
  • For non-validated surveys, consider soliciting 2-3 expert reviewers to provide feedback re: survey content and format.
  • If pre/post test collects indirect data (e.g., student perceptions), consider a plan to triangulate these data with a different source.

Onlooker/Participant Observations

Description Specific, systematic observations conducted to collect behavioral data about participants within a teaching or learning context. In onlooker observations, the observer is not a part of the intervention/ experience. In participant observation, observers are active participants in the intervention/ experience.
Data Potentially Yielded Data collected from trained observers will quantify or describe the behaviors of participants at one or at multiple time frames. These data might be tallies of observed behaviors, descriptive notes describing behaviors, or time-managed tracking of behaviors in an environment.
Quantitative Analysis Possible? Yes
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • All observers should be carefully trained to collect data that reflects the intent of the study.
  • Observations can be made in real time or via videotaped sample.
  • Consider gathering inter-rater reliability data if more than one reviewer is operating within the context or project.

Course Assignments/Projects/Assessments

Description Course assignments, projects, or assessments are any tasks students complete as a part of your class which can be used to understand participant mastery of content or performance at a given point in an academic term or program. This might include: writings, journals, projects, online assignments, quizzes, tests, etc.
Data Potentially Yielded Data reflects a wide array of possibilities, but commonly would reflect participant knowledge and/or understanding of course content at a specific point in time during the course’s duration.
Quantitative Analysis Possible? Yes
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • Any artifact that is part of a course you regularly teach or a course you have taught in the past can be used in a SoTL study. Consider the use of archival data to compare groups with and without a particular intervention or experience.
  • If you can no longer obtain consent from past students as they are gone from campus or you lack contact information for them, you can ask your IRB for a waiver of informed consent, so long as you have a plan to protect participant identity.

Written Student Reflections

Description Written student reflections are comprised of student thoughts and ideas presented that are expressed to demonstrate deep thinking and consideration (e.g., reflective journals).
Data Potentially Yielded Generally, data are journal entries or responses to specific reflection questions. Written data is analyzed to identify changes or trends across study participants.
Quantitative Analysis Possible? Yes, but less frequent than qualitative analysis
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • Reflection data is almost always derived from some sort of prompt (e.g., journal prompt, reflective question). Craft these prompts carefully to ensure that you’re collecting the data most valuable for your study.
  • Analysis of written reflection data is almost always a qualitative endeavor. There are a variety of valid approaches to this sort of work, so consulting with a qualitative researcher if this is a new form of method for you is a good idea.

Visual Student Reflections

Description Visual student reflections are comprised of student thoughts and ideas presented that are expressed to demonstrate deep thinking and consideration (e.g., concept maps, drawings, figures, photos).
Data Potentially Yielded Visual reflection data provide representations of knowledge, skills, or learning at a given point in time to identify changes or trends across study participants.
Quantitative Analysis Possible? Yes
Qualitative Analysis Possible? Yes
Miscellaneous Thoughts on Data Type
  • Visual reflection data is almost always derived from some sort of classroom project or activity. Craft these experiences carefully to ensure that you’re collecting the data most valuable for your study.
  • Consider various visual data analysis methods as a lens for understanding your data.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s