The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…


Leave a comment

Pedagogical Content Knowledge, Language Awareness, and edTPA: ISU Language Teacher Candidates’ Feedback Practices

Written by: Susan Hildebrandt, Ph.D., Associate Professor of Languages, Literatures and Cultures at Illinois State University

feedbackWhat makes feedback effective for students and encourages their later learning? Few would argue that timely feedback is desirable, and that it should provide students with a path forward. In the language classroom, the language in which the feedback is expressed also matters. My study, supported by a 2017 SoTL Summer Mini Grant, focused on ISU Spanish student teachers’ linguistic choices as they provided feedback to their K-12 Spanish learners during student teaching. Using the world language edTPA assessment task, I compared high-scoring edTPA portfolios to low-scoring portfolios. With an eye to similarities and differences in demonstrations of language awareness, the study investigated whether participants used English and/or Spanish in their feedback, along with the quantity and characteristics of that feedback.

edTPA became consequential in Illinois in September 2015, and all teacher candidates in Illinois who wish to earn a teaching license must pass that high-stakes, standardized assessment. edTPA evaluates teacher candidates’ ability to plan, instruct, and assess K-12 student learning through an extensive portfolio submitted to Pearson for external scoring at a cost of $300. It is a graduation requirement in ISU teacher education programs and is used in a number of states to evaluate effectiveness of teacher education programs.

Pedagogical content knowledge (PCK, Shulman, 1987) develops over the course of a career, beginning during preservice coursework and continuing throughout later in-service teaching and learning experiences (Henze & Van Driel, 2015; Lortie, 1975). It is domain- and discipline-specific (Hashweh, 2013; Shulman, 1987), comprised of content knowledge, or the what of teaching, and pedagogical knowledge, or the how of teaching (Shulman, 1987). Language awareness, as defined by Thornbury (1997), is “the knowledge that teachers have of the underlying systems of the language that enables them to teach effectively” (p. x). Language awareness, a subset of teacher candidates’ PCK, proves critical in language teacher education courses.

Much time is spent in postsecondary world language pedagogy classes to teach future language teachers how to teach in the target language, or the language that is being taught (i.e., Spanish) (Hildebrandt & Swanson, 2016). That training prompts teacher candidates to use the target language in a way that promotes student uptake and develops their language proficiency. It also seeks to avoid teacher candidates reverting back to teaching the way that they were taught (Cruickshank, Metcalf, & Bainer Jenkins, 2009); that is, it seeks to help teacher candidates avoid teaching about languages instead teach in the language itself. Methods classes can help teacher candidates internalize constructivist teaching practices and apply them to their feedback practices (Sigler & Saam, 2006), but previously held dispositions can prevent that application of best practices (Cummins & Asempapa, 2013). The language chosen for student feedback is critical in second language teaching, and scaffolding teacher candidate feedback to students can provide a valuable aid in long-term changes in teaching practices (Hunzicker & Lukowiak, 2015).

Both qualitative and quantitative methodology were used in this study. The initial data set was composed of World Language edTPA scores from a ISU’s seven Spanish teacher education program completers during the 2015-2016 academic year. Analysis of participants’ assessment artifacts (e.g., rubrics, evaluation criteria, etc.) allowed easier comparison of the level of class, type of rubric used, rubric criteria, feedback type, language of the rubric and feedback, among other features. Using qualitative analysis, I also explored ways that the portfolios with high average assessment task subscores manifested language awareness, as compared to the portfolios with low average subscores. Three general categories, based on the portfolios, formed the initial structure for analysis: language awareness, knowledge of L2, and knowledge of learners. Results showed that most participants’ feedback was in English, rather than Spanish. In general, those participants who provided more feedback to students, no matter the language, scored better on the edTPA assessment task.

This project sought to explore ways of preparing ISU teacher candidates of Spanish to complete an edTPA portfolio to align with communicative language teaching practices, including using the target language nearly exclusively. Those practices are explored and put into practice during coursework, but do not seem to generalize to the edTPA portfolio constructed during the student teaching semester (Swanson & Hildebrandt, 2017), as was found in the present study. I will use the information gained to create more opportunities for ISU teacher educators to create effective, Spanish-language feedback to their K-12 students (Hunzicker & Lukowiak, 2015).

The full study can be found in a chapter called “Pedagogical Content Knowledge, Language Awareness, and edTPA” that will appear early next year an edited volume. Researching edTPA Promises and Problems: Perspectives from English to Speakers of Other Languages, English Language Arts, and World Language Teacher Education will be released in early 2018 by Information Age Publishing.

Blog References:

Cruickshank, D. R., Metcalf, K. K., & Bainer Jenkins, D. (2009). The act of teaching (5th ed.).New York, NY: McGraw-Hill Higher Education.

Cummins, L., & Asempapa, B. (2013). Fostering teacher candidate dispositions in teacher education programs. Journal of the Scholarship of Teaching and Learning, 13(3), 99-119.

Hashweh, M. (2013). Pedagogical content knowledge: Twenty-five years later. In C. J. Craig, P. C. Meijer, & J. Broeckmans (Eds.), From teacher thinking to teachers and teaching: The evolution of a research community (pp. 115-140). Bingley, UK: Emerald Group Publishing.

Henze, I., & Van Driel, J. H. (2015). Toward a more comprehensive way to capture PCK in its complexity. In A. Berry, P. Friedrichsen, & J. Loughran (Eds.), Re-examining pedagogical content knowledge in science education (pp. 120-134). New York: Routledge.

Hildebrandt, S. A., & Swanson, P. (2016). Understanding the world language edTPA: Research-based policy and practice. Charlotte, NC: Information Age Publishing.

Hunzicker, J., & Lukowiak, T. (2015). Engaging pre-service teachers –and their professor – in learning: A comparison of two literacy methods courses. Journal of Transformative Learning, 3(2), 52-83.

Lortie, D. (1975). Schoolteacher: A sociological study. Chicago: The University of Chicago Press.

Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57, 1-22.

Sigler, E. A., & Saam, J. (2006). Teacher candidates’ conceptual understanding of conceptual learning: From theory to practice. Journal of Scholarship of Teaching and learning, 6(1), 118-126.

Swanson, P. & Hildebrandt, S. A. (2017). Communicative learning outcomes and world language edTPA:  Characteristics of high-scoring portfolios. Hispania, 100, 331-347.

Thornbury, S. (1997). About language. Cambridge: Cambridge University Press.

 

 

 

Advertisements


Leave a comment

Photo Documentation: SoTL Methods Series #4

Written by Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences and Disorders at Illinois State University

Over the last five years, students from my academic department have traveled to six countries on two continents as part of our program’s study abroad experience. Offered for independent study credit, these study abroad experiences have transpired as “short term” opportunities (e.g., spring break or two weeks in the summer session) and have focused on cultural immersion, rather than disciplinary content knowledge. I led our most recent trip to Spain during spring break 2017 and was joined by two faculty colleagues and 32 students. Prior to this year’s trip, students self-reported gains in what Miller-Perrin and Thompson (2014) would consider internal learning (e.g., emotional, intellectual, and spiritual growth) as well as external learning (e.g., second language acquisition, intercultural learning, globalization, disciplinary knowledge). While this was lovely to hear, these impressions were anecdotal, as we had never systematically studied student learning that occurred as a result of engagement as a study abroad participant.

I decided to study this year’s students to measure the impact of short term study abroad for this cohort. Initially, I thought I would restrict my study design to traditional methods of reflection and content analysis. In the weeks prior to travel, I asked students to set three goals for themselves based on self-perceived areas of weakness or interest. Students were asked to keep track of their growth in these goal areas via reflective journals, which were analyzed carefully for evidence of progress or growth in their goal areas. Eighty-one of these goals fit into one of the following categories: taking chances, engagement, flexibility, gaining independence, archiving, budgeting, social interaction, or understanding culture.

Around the time that I was developing the plan for collecting data from travelers, I read about photo documentation, a method derived from visual sociology wherein researchers seek out patterns in photographic data collected using something called a shooting script. I was intrigued as to how I might introduce this new method into my study abroad project. I tried, and I was able to extend what I learned about my students in doing so. The remainder of this post is a description of my “rookie experience” with photo documentation. I am certainly not an expert, but I did enjoy exploring this visual methodology!

Photo documentation is a research method developed by Charles Suchar (1997). Specifically, “photo documentation is a method that assumes photographs are accurate records of what was in front of the camera when its shutter snapped – ‘a precise record of material reality’ — and takes photographs in a systematic way in order to provide data which the researcher then analyzes” (Rose, 2016, p. 310). The key to photo documentation is a shooting script which consists of “lists of sub-questions” (derived as topical to overarching research questions) which act as a guide for taking pictures connected to the topic of the research being conducted (Rose, 2016, p. 311). Photos are taken in accordance with a shooting script, then analyzed for categories and patterns via a systematic coding process.

In the case of the study I described above, I wanted to understand changes that occurred in students in self-identified areas of need/interest. Thus, in addition to reflective (written) journaling, I asked that students use the following shooting script to document learning visually before and during travel:

  1. What was I doing/seeing when I recognized that I had made progress toward meeting one of my goals?
  2. What things/people/experiences influenced this progress?

spain food 2In applying this shooting script, students were directed to take pictures to answer these questions as they were going about their study abroad experiences. After travel, they submitted three photos per goal to me, along with their written journals. I printed out all photos that students submitted and searched for patterns across goals and photos. I was able to find many! As an example, many students set goals to improve their understanding of different cultures. Looking at the photos submitted along with written reflection for this goal, I could see that students represented “increases in cultural understanding”  via photos of food (see right), religious symbols, architecture, or Spanish citizens they interacted with during their travels. Students who set goals to “take chances” maggierepresented growth in this area with photos of new foods they tried or photos of experiences where they conquered personal fears (e.g., communing with the macaques in Gibraltar after a prior bad experience — see left).

Overall, what I found interesting is the different stories these data told. While written reflections told me WHAT changes students had realized as a result of study abroad, the photographs told me HOW how these changes occurred. The combination of these different data types were powerful to tell the story of my students’ experiences in Spain and made a powerful case for the possibility of significant learning in a short amount of time.

I have always been intrigued with what the visual representation of learning might look like for different students. Using photo documentation helped me to see the possibilities of visually-based data in a way that I appreciate and hope to use again in the future!

Blog References:

Miller-Perrin, C. & Thompson, D. (2014). Outcomes of global education: External and internal change   associated with study abroad. New Directions for Student Services, 146, p. 77-89.

Rose, G. (2016). Visual methodologies: An introduction to researching with visual methods (4th ed.). Sage: Los Angeles.

Suchar, C. S. (1997). Grounding visual sociology research in shooting scripts. Qualitative Sociology, 20(1), 33-55.

 


1 Comment

Think Alouds: SoTL Methods Series #3

Written by Sarah M. Ginsberg, Ed.D., Professor of Speech-Language Pathology at Eastern Michigan University (sginsberg@emich.edu)

Editor’s Note: This blog was originally posted on the SoTL Advocate on October 12. 2015 and is reprinted in its entirety now for its excellent fit in the current methods series which features methods for SoTL that are “new and different” to many.  — JCF

Screen Shot 2017-11-13 at 8.25.05 PMA common thread that runs through various cross-disciplinary SOTL research is the concerted effort made to understand what the accomplished professional is thinking when she solves a work problem so that we can use that knowledge as teachers to better prepare future professionals. That problem might be how a mathematician completes a technical calculation, or in clinical fields, it might be how the clinician arrives at a diagnosis. The value for all of us in understanding what our accomplished colleagues do in their heads when faced with a technical problem is that in identifying how the pros do it, we can uncover insights into how we should be teaching our students to think and to problem solve. This type of understanding relies on a process of collecting data while the person is actively engaged in solving a problem out loud. This type of study is often referred to as a think aloud (TA) and can yield important information to inform evidence-based educational practices

The TA method is a validated method of learning about cognitive processes by having participants verbalize their thinking in a metacognitive manner (Ericcson & Simon, 1993; Wineberg, 1991). TAs were popularized by Wineberg (1991) in his ground-breaking study that examined the differences between how academic historians processed information while reading historical texts and how students processed information regarding historical texts.  Since then, TAs have been used to study how novice thinking compares to experienced thinking in a wide variety of disciplines, including the health sciences, mathematics, and political science (Banning, 2008; Bernstein, 2010; Forsberg, Ziegert, Hult, & Fors, 2013; Wainwright & McGinnis, 2009). These types of studies are often referred to as “expert-novice” studies (Bernstein, 2010).

The process of data collection using a TA approach is quite simple and requires minimal technology and cost. Typically:

  1. Study participants are presented with the problem to be solved by the researcher and asked to solve it aloud.
  2. Specific directions are provided to participants. Prompts (e.g., “tell me how you would solve this” or “describe how you would approach this problem”) are used to elicit responses and gather additional information if a participant falls silent or struggles with the process.
  3. Participant responses are recorded for subsequent transcription and analysis.
  4. Once the TA is transcribed, the most challenging part of the process becomes the subsequent data analysis. Consistent with qualitative methodology, verbalizations may be read as a whole to determine initial emerging codes and impressions about the thought process (Creswell, 2002; Denzin & Lincoln, 2012). Using an inductive approach to identifying specific thought processes or strategies allows the researcher to move forward to developing secondary, axial coding. Themes emerge as the iterative process expands to include all participants and commonalities and differences can be appreciated.

Having recently completed a study comparing the diagnostic problem-solving of experienced speech-language pathology (SLP) clinicians compared to the problem-solving of SLP graduate students, I found that the most challenging aspect of analyzing the data was determining the level of thinking to focus on. I used studies in comparable clinical professions, such as nursing, physicians, and physical therapists to identify frameworks that might be useful to me. In determining the focus of my study, I chose to concentrate on the heuristics (thinking strategies) of my participants, to understand differences in approaches to problem solving and to create a framework that fostered comparisons to previous literature, potentially increasing the value of my findings.

For more details on the think aloud method and some outstanding examples of its use in a variety of fields, see the items included in the following references. It should be noted that a number of authors also advocate for the use of TA as a teaching method. For those unfamiliar with qualitative research methodology, several references are included here as well.

References for Additional Information on Think Alouds:

Banning, M. (2008b). The think aloud approach as an educational tool to develop and assess clinical reasoning in undergraduate students. Nurse Education Today, 28, (1), p. 8–14. doi: 10.1016/j.nedt.2007.02.001

Bernstein, J. L. (2010). Using “think-alouds” to understand variations in political thinking. Journal of Political Science Education, 6(1), p 49-69. doi:10.1080/15512160903467695

Ericcson, K. A., & Simon, H A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

Forsberg, E., Ziegert, K., Hult, H., & Fors, U. (2013). Clinical reasoning in nursing, a think-aloud study using virtual patients-A base for innovative assessment. Nurse Education Today, http://dx.doi.org./10.1016/j.nedt.2013.07.010

Wainwright, S. F., & McGinnis, P.Q. (2009). Factors that influence the clinical decision-making of rehabilitation professionals in long-term care settings. Journal of Allied Health, 38(3), 143-51.

Wineberg, S. S. (1991) On the reading of historical texts: Notes on the breach between school and academy. American Educational Research Journal, 28(3), 495-519.


Leave a comment

#Collaborative Auto-ethnography: SoTL Methods Series #2

Written by: Catherine McConnell (University of Brighton), Elizabeth Marquis (McMaster University), and Lucy Mercer-Mapstone (University of Queensland) — note complete author-supplied affiliations and contact information at the end of this blog post. 

When we (Catherine, Beth, and Lucy) met at the International Summer Institute on Students as Partners in 2016 we quickly discovered that, as practitioners of student-staff partnerships, we had many shared experiences. Our discussions were fruitful in terms of giving us a sense of belonging but we felt they warranted deeper exploration. So, we embarked on a process of delving into our own experiences in the hopes of learning in the process and sharing that learning with fellow practitioners and researchers. As we worked to find a way to effectively and systematically study ourselves, we arrived at the idea of using collaborative auto-ethnography as a methodological approach.


Above: Participants at the 2016 McMaster Summer Institute on Students as Partners in Teaching and Learning, where we first developed the idea for our CAE project.


Collaborative auto-ethnography (CAE) is a research method that involves ‘a group of researchers pooling their stories to find some commonalities and differences […] to discover the meanings of the stories in relation to their sociocultural contexts’ (Chang et al, 2013, p. 17). As such, it provided a perfect, if initially somewhat unfamiliar, way to collectively explore our individual experiences in a scholarly fashion.

We have put together this blog post to explain our developing understanding of the method and process of CAE, and how we, as a group of three researchers, have used it in our recent SoTL enquiry into ‘student-staff partnership in higher education.’

CAE builds upon ‘auto-ethnography,’ which is a method that uses a researcher’s personal experience to ‘describe and critique cultural beliefs, practices, and experiences’ (Adams et al, 2015, p.1). Auto-ethnography is a deeply personal and reflective process, usually taking place in a researcher’s own context, whether that be about practice or in a certain situation. Adding the dimension of multiple ethnographies (i.e., more than one voice) presents the possibility that the method can simultaneously be collaborative, autobiographical, and ethnographic.

For our shared research, we took up the CAE method through structured reflective writing, designing a writing activity to provide a framework that would guide our individual reflections. We set ourselves a word limit of 750 words that we would write and share with each other in a private online space. This activity was modelled on a set of reflective prompts, informed by Johns’ model of structured reflection (Johns, 2000), which had been adapted by colleagues at the University of Brighton, UK (2011) and which we subsequently re-appropriated. Specifically, we:

  1. Developed a framework of prompts/questions for reflection
  2. Wrote individual reflections guided by the framework
  3. Shared and read each other’s reflections
  4. Conducted iterative thematic analysis to discover key themes

Our developed framework was a useful facilitative tool to prompt us to take an open, honest account of our personal and professional experiences, affirming Johns’ interest in ‘making explicit the knowledge we use in our practice’ (Jasper, 2013, p.86). Posing questions that follow Johns’ (ibid, p.37) format of phased cue questions (phases involve preparatory, descriptive, reflective, anticipatory, and insight questions) we focused our SoTL enquiry on identity construction, navigation and enactment in the context of student-staff partnerships.

The framework below provides an illustration of the types of questions used to prompt our personal reflections about identity in the context of student-staff partnerships. This could be easily adapted to other SoTL topics –  especially those that reflect on practice.

Framework for reflective questioning
Description of experience Phenomenon: describe in detail your partnership practice, or a specific partnership experience that seems especially noteworthy, without interpretation or judgement
Context: what were the significant background factors to this experience? Why did it take place, and what was its purpose?
Reflection What were you trying to achieve?
Why did you behave as you did?
What were the consequences of your actions for yourself and others?
How did you feel about the experience when it was happening?
To what extent did your actions realize your understanding of partnership?
What identity(s) were you consciously aware of at the time?
What identity(s) do you believe were at play during this interaction in hindsight?
If multiple, which identity was most salient? How were they interacting?
Analysis: Influencing factors

 

What factors influenced your decision-making? Some potential options to consider: Prior experiences, Societal expectations/ideologies/assumptions, Context
How was your salient identity affecting your actions?
How was the interaction between identities affecting your actions?
How was the presence of this identity(s) influencing your perceptions of those with whom you were interacting?
Analysis: Alternatives What other choices did you have?
What could be the consequences of these choices?
Learning & Action How do you feel about this experience now?
Has this experience changed your way of understanding yourself?
Did your salient identity change? If so, how and why?
In hindsight, how has is interaction/even affected your ongoing identity in partnership?
What new questions, challenges or issues has it raised?
Given the chance, what would you do differently next time?
How will you follow up on this experience in order to put your learning into practice?

Once we had written and shared our individual reflections, we found it useful to read each other’s, and write a short ‘meta-reflection’ on the writing process. This enabled us to appraise the CAE method straight after the reflective process but before we began any analysis of the transcripts. Some of our observations included:

  • The researchers felt a sense of belonging and solidarity to one another along with a communal ownership of an enquiry
  • Writing and sharing reflections caused each of us a personal realisation and provided grounds for transformation. This process was not without some discomfort, though, perhaps because there is a level of vulnerable-making involved with sharing personal reflections with colleagues
  • Sometimes the content of the reflections themselves was also unsettling or challenging, and caused discomfort in relation to an aspect of one’s own practice
  • We experienced a heightened consciousness of our own values and beliefs relating to practice, the influence these have in partnership situations, and in our expectations of others
  • We also noted a sense of excitement when reading each other’s writing inspired by the experience of sharing personal insights and aspects of our own identity that are usually implicit

While CAE proved a useful method for meaningfully exploring our research questions about our own experiences, then, it was also an exciting and sometimes uncomfortable process that supported reflective thinking and potential development of our practice as teachers and learners.

Project Information

We are three SoTL practitioners working in the area of student-staff partnership in HE across three western countries, in differing roles, and in significantly different institutional contexts. Catherine McConnell is a Senior Lecturer in a Learning and Teaching centre, focusing her work and doctoral research on student-staff partnership, at the University of Brighton in the UK. Beth Marquis is Associate Director (Research) at the central teaching and learning institute at McMaster University in Canada. Her disciplinary training is in film and cultural studies. Lucy Mercer-Mapstone is a PhD candidate and research co-fellow at the University of Queensland, Australia. She currently leads the collaborative design of a university-wide Students as Partners program that aims to embed a culture of partnership at the institution.

You can find out more about our study: Breaking Tradition Through Partnership: Navigating Identities and Dissonance in Student-Staff Partnerships in the EuroSoTL Conference Proceedings, p296.

Catherine McConnell * a, Elizabeth Marquis b, c Lucy Mercer-Mapstone

a Centre for Learning and Teaching, University of Brighton, Sussex, England. C.McConnell@brighton.ac.uk

b Arts & Science Program and MacPherson Institute, McMaster University. Hamilton, ON, Canada. beth.marquis@mcmaster.ca

c Institute for Teaching and Learning Innovation, University of Queensland, St Lucia, Brisbane, 4072, Australia, l.mercermapstone@uq.edu.au, orcid.org/0000-0001-7441-6568

*Corresponding author

 

Blog References

Adams, T. E., Linn, H. J. S. and Ellis, C. (2015). Autoethnography. Oxford: Oxford University Press.

Chang, H., Ngunjiri, F., & Hernandez, W. (2013). Collaborative autoethnography. London: Routledge.

Jasper, M. (2013) Beginning Reflective Practice. (2nd edition) Hampshire: Cengage Learning.

Johns, C. (2000) Becoming a Reflective Practitioner. Oxford: Blackwell Scientific Publications.

University of Brighton. (2011). Critical Incident Analyses. Brighton: University of Brighton. 


Leave a comment

Theoretical Pattern-Matching in SoTL: SoTL Methods Series #1

This blog serves as the beginning of a four-week focus on unique research methods for SoTL work. Enjoy, and please feel free to write to our guest bloggers with any feedback or questions! -Jen Friberg, blog editor

Written by: Bill Anderson, Associate Professor of Family and Consumer Sciences at Illinois State University (jander2@ilstu.edu)

I recently completed a SoTL project where I was hoping to create vicarious, but meaningful, applications of classroom learning, in this case, foundational theories of the human development field. In an attempt to accomplish this, I utilized interrupted case studies (ICS), a progressive disclosure of information viewed as problem-based learning over time. Over an eight week period following a pre-test application, students viewed a longitudinal series of interviews as an ICS. This series followed several participants from the time they were seven years old in 1964, revisiting them every seven years until age 56 in 2013. During the process, and using the assumptions, concepts, and language of assigned developmental theorists, students described and applied relevant theoretical positions to anticipate growth and change as this collection of real lives progressed. Their work was submitted in weekly reflective essays. At the end of the eight-week assignment, post-test results indicated that the method was quite successful but told me nothing further. The post-test increase could simply be the result of memorizing the material. Therefore, pattern-matching was applied to further examine those results.

patternPattern-matching is a less-known, but dependable, procedure for theory testing with case-studies and is regularly recommended for reconciling mixed methods and data sources in case study research, and to boost the rigor of the study. The overarching goal is to explain relationships between key points, in this case the pre/post results, by comparing an identified theoretical pattern with an observed pattern.

The previously mentioned weekly student essays were utilized as the observed pattern. These included descriptions of their assigned interview participants, appraisals of their most recent developmental predictions for this person, and their expectations for the next seven years. The essays were coded line-by-line to determine the degree of matching to a predetermined theoretical pattern. In this case, Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001) was utilized as coding categories as follows: 1 – Remembering, 2 – Understanding, 3 – Applying, 4 – Analyzing, 5 – Evaluating, and 6 – Creating. Average use of these levels could show a general progression from simple remembering (e.g. defining, telling, listing) to application (identifying, selecting, organizing) to creating (imagining, elaborating, solving). Once the essays were coded, interrater reliability was determined by using the intra-class correlation coefficient function of SPSS v. 20 to determine a kappa score of reliability, with a score of .80 deemed reliable.

Results, the observed pattern, allowed me to see a progress toward more complex reasoning in the assignment as the class progressed and students gathered more information and became more comfortable with theory application. Briefly stated, the first essays indicated an average response at Bloom’s applying level. Students were identifying and correctly applying concrete elements of the theories and making tentative, but informed, inferences. However, by the final essay the average response level was solidly at the evaluating level. There, students were appraising the flexibility of the theories being applied along with the documentary participant they were following. It became more common to see students suggest multiple possibilities in their writing, prioritize these, and determine the most informed interpretation. Consequently, pattern-matching indicated an established theoretical progression in reflective thinking from pre- to post-test.

Still, very few specific examples of best-practice exists with pattern-matching (Almutairi et al, 2014) and applications in SoTL (and education, in general) are rare. However, there are a number of available theories that could be considered as an identified pattern. For instance, I am currently using William Perry’s (1999) scheme of intellectual develop during the college years as a pattern basis in order to better understand contemporary student’s willingness, or unwillingness, to discuss racism in the classroom. Perry’s scheme is noticeably related to Bloom’s work, though somewhat better suited to assess student readiness to learn. Lastly, there are several other established variations of pattern-matching. In you are interested, a good place to begin would be Robert K. Yin’s (2009), Case study research: Design and methods.

References

Almutairi, A.F., Gardner, G.E., & McCarthy, A. (2014). Practical guidance for the use of pattern-matching technique in case-study research: A case presentation. Nursing and Health Sciences, 16, 239-244. doi: 10.1111/nhs.12096.

Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Blooms taxonomy of educational objectives. London: Longman, Inc.

Perry, W. (1999). Forms of ethical and intellectual development in the college years: A scheme. San Francisco: Jossey-Bass.

Yin, R.K. (1994). Case study research: Design and methods (2nd Ed.). London: Sage.

 

 

 


Leave a comment

Making the Case for SoTL Self-Advocacy in Academic Job Searches

Written by Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor, Communication Sciences and Disorders at Illinois State University

Two close friends are currently on the job market. Presently employed as associate professors (or close to that rank!) each has chosen to seek novel adventures elsewhere in academia. Thus, they are in the process of sifting through position descriptions and polishing their teaching and research statements to submit in the coming weeks, following a course familiar to many in higher education.

search1Each of my friends have been successful disciplinary researchers as well as productive SoTL scholars, though they both represent disciplines that do not consistently value SoTL. As they contemplate phone interviews and campus visits in the near future, they have wondered aloud about how they might contextualize their SoTL work in a way that positions them well in their job searches. While it’s disappointing to me that they have to consider this issue (SoTL should be uniformly valued!), I recognize it’s likely very necessary and, in fact, is smart preparation for their respective job searches.

With that in mind, the contextualization my friends seek as a framework for their SoTL work could actually be a form of SoTL self-advocacy, which I’d define as anything a person does to describe the value of their SoTL work to relevant stakeholders. SoTL self-advocacy might look different across contexts, but in the milieu of a job search, there are definitely steps my friends could take to share the appeal and impact of their SoTL work. Specifically, I would advise each (and would tell others!) that they might do the following to engage in SoTL self-advocacy:

  • Closely examine the mission/vision statements for any institution of interest. Mine these statements for alignment with past and current SoTL projects. Prepare an explanation for how your own SoTL work meshes with or could advance the mission/vision of the institution of interest.
  • Read the strategic plan carefully, noting where your SoTL work matches with current initiatives/efforts being undertaken by your institution of interest. Seek out mentions of student involvement in research, evidence-based instruction, or broad definitions of scholarship. Use these to frame your SoTL work and plans for the future in the interview process.
  • Reflect on the impact of your SoTL work. Many classroom-based SoTL studies lead to changes in curriculum, teaching methods, course design, etc. Be prepared to discuss how SoTL has impacted your teaching or your students’ learning. Have there been impacts to your current department (e.g., curricular changes) that resulted from your SoTL work? These would be important to describe.
  • Closely examine the vitas (hopefully archived online!) of faculty in the department/school/unit you might be looking to join. Determine if any individuals are SoTL-active or could serve as collaborators in the future. Be prepared to talk about how you would engage colleagues in institution-relevant collaborative projects.
  • Visit websites for the teaching and learning center, provost’s office, and/or any other important campus units affiliated with your institution of interest. Look for prompts that support SoTL and precedents for SoTL engagement. Identify individuals on campus who might be in the position to discuss SoTL opportunities at the institution and (if appropriate) attempt to have them included in your on campus interview (or express interest in meeting them if they are not in your interview).

Other ideas for SoTL self-advocacy on the job search? Please post below in the comments section!


Leave a comment

How Are We SoTL-ing?

Written by: Jennifer Friberg, Cross Endowed Chair in SoTL and Associate Professor of Communication Sciences and Disorders at Illinois State University 

In the run-up to ISSoTL 2017 last week in Calgary, Alberta, Canada, it might have been easy to miss that the latest issue of Teaching and Learning Inquiry (TLI), the journal of the International Society of the Scholarship of Teaching and Learning, has just been published. I had the opportunity to read several articles in this issue prior to traveling to the conference and was particularly interested in one article, Survey of Research Approaches Utilised in the Scholarship of Teaching and Learning Publications, which was co-authored by Aysha Divan (U. of Leeds), Lynn Ludwig (U. of Wisconsin-Stevens Point), Kelly Matthews (U. of Queensland), Phillip Motley (Elon U.), and Ana Tomljenovic-Berube (McMaster U.).

Why the interest? As a SoTL faculty/student developer, I am forever asked if there is a “preferred” method for engaging in SoTL. I have always addressed this topic from an anecdotal perspective, simply telling novice SoTL scholars that qualitative, quantitative, and/or mixed methods are all equally appropriate for SoTL, depending on the “fit” of the method to the study aims/design. With this paper, a bit more clarity was offered as a result of systematic study of three years of published SoTL journal articles.

Honestly, I imagined that there were far more qualitative methods employed in SoTL research than quantitative; however, I was incorrect. Overall, 223 articles from the following journals were studied: International Journal for the Scholarship of Teaching and Learning, the International Journal for Teaching and Learning in Higher Education, and the Journal of the Scholarship of Teaching and Learning. Across these articles, there was an almost even balance of quantitative, qualitative, and mixed methods research (see graphic below).

methods breakdown

Of even greater surprise to me were the following findings:

  • 84% of papers utilized a single data source for reporting (primarily students), which leaves the need for triangulation of data open for consideration in terms of future project planning.
  • Data from mixed methods studies were often times poorly integrated with only 30% of studies fully integrating qualitative and quantitative data as part of the discussion of findings.
  • 65% of studies relied on a single “snapshot” of data (data collected at one time only), which leads to thoughts on the value of/need for collecting longitudinal data to study student learning over time.

At ISSoTL last week, Gary Poole delivered a plenary address reminding us all that as professionals interested in SoTL, we have a choice to facilitate or hinder as we collaborate and mentor. As a professional developer for faculty and students interested in SoTL, I intend to share this information as a facilitative effort to grow SoTL at ISU (and beyond), helping future SoTL scholars to be mindful of trends, needs, and considerations in SoTL publishing. Specifically, I will urge SoTL researchers to:

  1. Seek out a “goldilocks” fit to connect their research questions to the type of data they collect. Why? This allows a researcher to determine whether research question(s) being posed are best answerable with qualitative, quantitative, or mixed methods approaches. A good fit is critical for a study to make sense to interested stakeholders.
  2. Ensure that data come from as many direct data sources as are necessary to form a strong foundation for any discussion of results/implications.
  3. Use indirect data sources primarily as support/triangulation for data collected from direct sources.
  4. Think carefully and critically about how data from a study is discussed. If the design selected has a mixed methods approach to data collection, then all aspects of data should be explored in an integrated manner to identify trends and accurately interpret and report data across the board.
  5. Consider whether data collected at multiple data points might be more appropriate for a study than a “one-time” data collection effort in order to best answer the research question(s) being posed.

 

Blog References:

Divan, A., Ludwig, L. O., Matthews, K. E., Motley, P. M., & Tomljenovic-Berube, A. M. (2017). Survey of research approaches utilised in the scholarship of teaching and learning publications. Teaching and Learning Inquiry, 5(2).