Inclusive Assessment: Find Out What You Might be Missing

Have you ever given an assessment that was not about content? While content may still be king, it is not in the castle alone. Join Dr. Allison Case and her guest, Dr. Christina Paguyo, Director of Academic Assessment at the University of Denver, as they explore inclusive assessment—the layering of non-content based assessments on top of traditional content assessments as a way to better serve and respond to our students so they can fully engage with our class and our content.

Show Notes
Dr. Virginia Pitts talks about having clarity about what you want students to walk away with when they leave your class five years from now, they're looking back and thinking about your class, what knowledge, what ideas, what feelings would you want to have endure? https://otl.du.edu/what-to-do-when-it-all-feels-new-teaching-at-a-social-distance/

Christina referenced the writings and practices of Rachel Toor, Professor at Eastern Washington University, in using low stakes, ungraded assignments to check in on students. You can read more about her work here:  https://community.chronicle.com/news/2367-turns-out-you-can-build-community-in-a-zoom-classroom

Christina referenced the book "Small Teachings Online: Applying Learning Science Online" by Flower Darby and James Lang. https://www.amazon.com/Small-Teaching-Online-Applying-Learning/dp/1119619092

Additional resources:
North Star reference: https://otl.du.edu/courses-opportunities/workshops/course-design-institute/
Stress test: https://otl.du.edu/mitigating-increasing-inequity-as-we-move-online/

For a transcript of today’s show, additional resources, or to listen to previous episodes, come visit us online at ExamSoft.com/Pedagogo. 

Transcript

Joe Taucher: (00:00)
Pedagogo the show that brings education to your ears and metamastery to your assessments. Today's episode covers inclusive assessments, with ideas on how to ace the most important test of all. Building assessments that go beyond the letter grade. Pedagogo brought to you by ExamSoft, the assessment software that keeps security and integrity in your exams while providing you actionable data for your outcomes. For all of the toughest testing challenges, ExamSoft has you covered. 

 Allison Case

It’s fall, education nation. Day by day we are making it through this semester. I hope you are taking time to appreciate your and your students' wins. This is a crazy time.  Today’s episode provides some insight into how to succeed in this chaos of 2020, not just despite it. It opened my eyes to a whole new world. Let me ask you this.   Have you ever given an assessment that was NOT about content? Beyond a “getting to know you” worksheet I gave on the first day of class, I never did. All of my assessments - formative, summative, high stakes, low stakes, were about content. And while content may still be king, it is not in the castle alone. Today’s episode is about inclusive assessment, or the layering of non-content based assessments onto our traditional content assessments  as a way to better serve and respond to our students so they can fully engage with our class and our content. 

 One of my first questions about inclusive assessment was “inclusive of what?” As you’ll hear my articulate  guest share with us today, to fully understand our students, we need to view them as their whole selves. And doing so will  span everything  from the physical to the emotional, to the cognitive, to the mental and  even spiritual. And so inclusive assessment provides us entry points to and illuminates opportunities for us to better understand where our students are in regards to these dimensions. If you’re already feeling overwhelmed. Just wait!  After listening today, you’ll realize that assessments can query much more than just content, and for many more reasons beyond measuring content knowledge. Assessments can open the doors to where students are, how they are doing, and what they might need to be there best.  The best news of all? You’ll hear Christina explain that this new approach to assessment doesn’t change everything about your class, and doesn’t have to take a lot of time. There’s so much to unpack today, so let’s get started. 

Allison Case (00:31)

Well, Christina, thank you so much for joining us today for our discussion on inclusive assessments to kick us off. What keeps you busy on a day to day basis?

Christina Paguyo (00:40)

Sure. Thank you for having me. It's really exciting to join you all. My name is Christina Paguyo and my official title is Director of Academic Assessment at the University of Denver, but I informally call myself an academic DJ who choreographs assessment and accreditation with the everyday activities of faculty, staff, and students.

 Allison Case (1:07)

Wonderful. Our first DJ. I love it. If you could define for us, what is inclusive assessment?

Christina Paguyo (1:12)

So, inclusive assessment is uh, an opportunity to create space for multi-voiced multi-dimensional aspects of learning. And so it's not just about querying for learning, or querying learning data. It's also about kindling, robust learning. So, it's multiple aspects thinking of inclusive assessment as a way to build relationships with students, to query, kindle, and then query and kindle again, redesign learning environments, uh, so that students and faculty are able to make the most of the educational experiences.

Allison Case (1:53)

So, it's multi-dimensional querying learning in a way that informs the student and the faculty member, in a way that allows response to that information, specifically that the faculty member can pivot in the way that they either relay the information, build upon it, query it in the future. While creating a relationship, and in a way that considers these multiple aspects.

Christina Paguyo (2:20)

Yeah.

Allison Case (2:22)

I have a follow up question for you about this, and I'll give you a second to think about it. We're talking about this being assessment that considers the multi-dimensional influences on, on a human's life. What do I look for to appreciate a considerate assessment versus an inconsiderate assessment, if you'll allow me to use that kind of terminology?

Christina Paguyo (2:44)

Yeah, for sure. So, this creates a throughline back to the philosophical hallmarks of inclusive assessment or what I think of as inclusive assessment. There are three organizing principles to consider. And the first one is to think about what is your approach to learning and with inclusive assessments, it is an outgrowth of learning sciences philosophy about positioning everybody as learners and everybody as experts. And so our students have a lot of expertise and I can learn from them. Okay. I also have expertise that they can learn from. And so together we are learning with and from each other. And then the second organizing principle, I think of, when it comes to inclusive assessment is establishing a North star. And so a learning scientist, Dr. Virginia Pitts talks about create, having clarity about what you want students to walk away with when they leave your class five years from now, they're looking back and thinking about your class, what knowledge, what ideas, what feelings would you want to have endure? And identifying that North star helps to create a compass to orient and design and redesign the class. And then the third organizing principle is to consider a constellation of transparency and flexibility.So, being as transparent as possible about here's what I'm doing and why, and also having elasticity flexibility to shift things instead of keeping to an original plan. So, those are some organizing principles. I just wanted to put out there to help orient ourselves to a why I would suggest more concrete, particular strategies.

Allison Case (4:55)

Yeah. What stands out to me, Christina is these three organizing principles have much more to do with the people and the learning in the classroom than it does assessment. And that's surprising. So, can you talk to us about how these guiding principles, um, relate back to your tie to inclusive assessments?

Christina Paguyo (5:19)

Absolutely. And so I, it depends upon our North star as educators. And I think too often in conversations about assessment, the goal is do assessment, do some kind of assessment, and then we check it off the list and it's done. And I'm saying instead of framing assessment, as the goal, I reframe it as the goal is using assessment as one of many tools to help us get to that North star where our students see themselves and they are positioned as designers and architects of their future and their lives. There is not a one size fits all. Everything has to be taken into context. So, how does that translate to right now? Um, if we're thinking about inclusive assessment, well, a professor first may want to conduct a stress test with the students to find out formatively, like, where are the students at? What is stressing them out, or do they have access to economic security, food, security housing? Do they have access to quality technology? What other stresses are happening? Um, and the reason why this is important to consider is because it will impact perhaps how the professor decides to organize the classroom. So she, he, or they might decide, “oh, okay, um, we have students who don't have the same kind of literal technological bandwidth as our other students. So, the additional digital platforms I thought that we could experiment with shouldn't necessarily be a requirement for the class”. Or, “oh, um, this student is experiencing food insecurities. I will refer them to the food pantry or somebody is going through grief and is, is, has lost a loved one to COVID-19. I will make sure or do my part to encourage them to go to the counseling center, whatever supports are.”

Allison Case (7:39)

The light bulb that just went off over my head is it's not enough to put in your syllabus, “we have counseling, we have a food pantry” because that puts it on our students to advocate for themselves. And what you're saying is we play an active role in advocacy for them by facilitating the collection of this information. 

Christina Paguyo (7:58)

Exactly. So, with inclusive assessment, the professor can serve as a bridge and as a mediator to connect students to multiple parts of their educational experiences, whether that is connecting them to each other, connecting to the professor, connecting to the academic content and connecting them to other resources that will support the student in their growth and development and learning.

Allison Case (8:26)

So, I taught a sophomore level engineering class called Statics. Big picture, when I walk away, what do I want for them for that class, it's to be strategic and feel confident.  So, how does, Christina, my desire for my students to be strategic and feel confident as an outcome of my class tie back to what you're trying to teach us about inclusive assessment?

Christina Paguyo (8:52)

Yeah. So, I think this is where we can become more concrete about shifting from the ontology and philosophy and principles of inclusive assessment and focusing on what are some concrete, micro, pivots, and actions that faculty can take in harnessing inclusive assessments at the classroom level. And so I would still recommend a diversity of formative and summative assessments. And the way that I think about formative assessment is I'm learning how to ride a bike. And I wish I had this when I was growing up, but I see these little ones these days with the push bikes have any pedals.

Allison Case (9:42)

Totally not fair. 

 Christina Paguyo (9:55)

And so I think of formative assessment as the push bike, like how it can help us respond to spontaneous concerns like, "oh, I'm going to fall if I don't put my feet on the ground, right?"

Allison Case (10:06)

Right.

Christina Paguyo  (10:07)

So, over time because of how the bike is engineered the kiddo, the little one learning how to ride will gain more confidence over time. Right. And then eventually we get to summative. So, if we were to use an expand upon that cycling example, the summit of assessment piece would be, depending upon the person, maybe simply writing a two wheeler and bicycle without any supports, uh, for someone else who might want to have more of a sensation that could be mountain biking or racing in a triathlon, um, and anywhere in between. And so that's sort of how I create like a, a metaphor between biking and formative and summative assessment. And so I believe my lovely colleague, Dr. Valentina Iturbe-LaGrave talked about a lot of formative assessment, which is key to inclusive assessment can also help paint a picture for the professor to remind you about the wholeness of our students. And if they are not in an emotional stable space, what kind of learning is going to happen? What kind of learning happens then assessment becomes moot if we just focus on the outcome without recognizing the context. 

Allison Case (11:51)

So, in this regard, it's almost humanity probing versus content probing.

Christina Paguyo (11:58)

Yes.

Allison Case (11:59)

That we can establish a recipe to collect information that will work in tandem.

Christina Paguyo (12:06)

Yes, absolutely. I see.

Allison Case (12:09)

When we say inclusive assessment, what are we including? Is it a ‘what’? is it a whom? Is it both?

Christina Paguyo (12:16)

Yes. And so when I conceptualize inclusive assessment, I think about wholeness and who are our whole students, what makes our students whole, and it spans everywhere from the physical to the emotional, to the cognitive, mental and spiritual. And so inclusive assessment provides us entry points and illuminates opportunities for us to better understand where our students are at, in regards to any of those dimensions. And also to turn that back on ourselves, to recognize where we might be at, in any of those dimensions all of this, co-constructs the landscape of higher education and education writ large. And so if there's anything I would want our listeners to take away from inclusive assessment is to think about the ways in which each person is a composite of a lot of different iterations and feelings and emotions and intellect that can shift from moment to moment and to only zero in on the intellectual or cognitive piece and content area, uh, makes and pretends that we're one dimensional when we're not.

Allison Case (13:52)

What are some of the stumbling blocks that get in the way of inclusive assessment?

Christina Paguyo (13:56)

Yes, there are a few, but I will highlight three. The first stumbling block, I believe, is fear. So, many professors and people in general are fearful of appearing like they don't know everything, and oftentimes steeped in sort of this, um, velvet niche of academic cynicism, like, “oh, okay, great inclusive assessment, another fad that we're gonna play with. And so my proposed solution to that stumbling block is to simply say, you don't have to change every single thing. Don't have to overhaul everything. And in fact, research shows that, uh, that, that doesn't have sustaining long term impact.

Allison Case (14:47)

Right.

Christina Paguyo (14:48)

And so what would be most supportive are micro-pivots. And so what are just one or two things you want to experiment with and try. So, maybe if, if you're interested in inclusive assessment, um, start out with a stress test, maybe, um, consider choosing one student learning outcome that is really important and valuable. Using that to sort of experiment with, "oh, okay, how do we scaffold the students who engage in weekly or regular reflections. I would recommend to preserve steady state in some way that a professor feels very strongly about like, this is the key stone of this class, and I'm not going to change this, but to be open and receptive to making micro pivots and shifts in other areas that could be further strengthened. The next stumbling block is privileging the tool over your North star. So, I'll give an example. Sometimes there are amazing tools to do assessment or even to do some kind of pedagogy and people will immediately bring that in to the classroom or into their program without taking time to critically reflect on, huh, does this actually help me meet my goal? Like how does this fit in with the actual class or the program and the goals that I have in mind. And so rather than privileging the tool, a solution to that would be taking time to consider and reflect on what is the North star.

Allison Case (16:38)

I love that because there is so much information and so many learning tools available to us. Otherwise it's just drinking from a fire hose. There's an app for everything. So, having new guiding North Star facilitate the way you view technology, um, and, and tools I think will be really helpful. That's great.

 Christina Paguyo (16:59)

One last stumping, stumbling block is when faculty may hear the word inclusion or diversity equity, and they automatically think, well, that has nothing to do with my discipline. So, I'm going to, uh, wash my hands of it. And, um, I would invite faculty think about, as a way to build relationships with students that then can help students build greater connection with the content that the professor's attempting to teach.

Allison Case (17:33)

Yeah. That makes so much sense. I'm glad you said that because I can understand why, you know, I think we're talking largely about like STEM disciplines, disciplines that don't have so much to do with humanity or the humanities. It can be easy to think, “well, when I teach, you know, Navier Stokes equation, it doesn't matter who or what you are. It is what it is”,  but yet you just make a really good point about how to layer in those activities, in those points of reflection on top of what is your unchanging content.

Christina Paguyo (18:08)

Yes, yes, exactly.

Allison Case (18:11)

So, Christina, what do I need to know to do this well?

Christina Paguyo (18:14)

Well, I would say first to look at some immediate strategies and tips that you can put to practice in the classroom. So, as I alluded to earlier, having a pre survey, a stress test is really helpful. So, you have a feel for how your students are doing from a more holistic perspective. Okay.

Allison Case (18:38)

And that would kind of give me my baseline.

Christina Paguyo (18:41)

Yes. It would give you your baseline. So, um, so in addition to the stress test, some check-ins that will help, um, create space for students to share of themselves if they feel comfortable. And so one of my favorite activities comes from Rachel Toor, who is a writing professor. She wrote about this in the Chronicle of Higher Education, but she has created community and online settings with a low stakes ungraded assessment. Um, and then another formative assessment, um, that is ensuring that you, as the professor provide feedback that is timely. That is so crucial, especially during this online setting where we're in a different setting, different norms, not sure how to interact with each other. And the only way we can really get cues about how well we’re doing is through feedback. And so I would recommend ungraded and graded forms of feedback that will allow students to build their confidence along the way, again, that notion of, uh, being a push bike so that they are growing stronger and in the mechanics and in engaging with the content. Um, so something that I love, which comes out of  Flower Darby and James Lang, which is a ‘Small Teaching Online’ book that I highly recommend. She talks about providing audio feedback. if students have submitted something, you can do an audio or video feedback highlighting the things that students did well and then opportunities for them to grow can also, And so for students to hear your voice or to see your face and your expressions along with that helps to make you more whole to them, as opposed to this invisible instructor who is creating assignments.

Allison Case (20:48)

Wow that's neat. Feedback is undeniably critical to the learning process. I mean, we receive feedback, you know, in our environments every day that allow us to understand how to operate within them. But if it is not the most time consuming element of teaching, you know, I don't know it is. And without kind of these tools or these new ways beyond long, you know, handwriting, everything, it just can feel like a tsunami coming at you every time assignments are due.

Christina Paguyo (21:20)

Yes, absolutely. So, in the spirit of practicing self love and self care, it is important to think about what are some ways in which we can make this process more efficient. 

 Allison Case

In what ways can technology play a role in building these inclusive assessments.

Christina Paguyo (21:34):

This is about helping to reveal any pain points that students and faculty are experiencing in or out of the classroom to see what can we do to better support our students. If I were a professor talking to a student and a student hasn't logged in within the first week of class, I would see that in the metrics of the LMS and I will contact the student and say, “hey, I noticed that you haven't logged in. I want to check in and see how you're doing. As you know, we are in a really rigorous class and it's important to hit the ground running. And I'm concerned that if you don't log in, we won't be able to support you in hitting the ground running. So, what can we do? Should we set up a time to chat?” You know, it's very much a way of building relationships with students. 

 Christina Paguyo (23:00)

It goes back to one of the organizing principles I talked about in regards to inclusive assessment is being immersed in this constellation of transparency and flexibility. And so making very explicit and clear like here is this tool that we're using. Here's how you can use it. Here's why we're using it. And here are its affordances and constraints. I mean, speaking of wholeness, right? Um, it's important, especially with technology. 

 An example is at ExamSoft, the software automatically can generate these strengths and opportunities reports . We advise, “hey, pull up an example one five minutes in class and say, “when I send this to you, here's how to interpret it. Here's what it means. And here's what I expect you to do with it”. Absolutely talk about how it will be used for what purpose of, maybe even spend some time talking them through the pieces. “This is the information that you can glean from this, if you, if you choose to use it.” So, I just think that goes a really long way.

 Christina Paguyo

Moved from 23:42, beginning - Absolutely. Absolutely. And, um, (Moved from 23:42 paragraph- end)There are two things I want to emphasize. Technology is not the goal. Technology is a tool to help us, um, get to the more expansive, robust notions of what we think education ought to be doing for our students.

Christina Paguyo (24:01)

So, any way that we can equip our students and our faculty with the ways in which tools can support and strengthen their learning in teaching, make that transparent, um, the better. And that, that reminds me of the second part that I wanted to talk about in. We want it to be a tool to get to student learning, um, is also combining technology with that human interpretation and that human touch to build a relationship. And so the tools that you've described Alison, um, from ExamSoft could help a professor, for example, build a conversation with a student like,  “hey, can we set up a time to meet here's what this technical tool is showing? Can you help me understand how you're interpreting this information? Like what are all the different ways that we can interpret this?” Um, and again, that is going back to an organizing principle of inclusive assessment, which is positioning everybody as learners and as experts. And by asking the student for his, for their interpretations, we are harnessing their expertise and then the professor can offer their interpretation and together co-construct okay, well, this is what the data says, that here's how we want to interpret it. And here's how we want to move forward.

Allison Case (25:31)

What a gentle way to be invited to a conversation, not a meeting in your instructor's office hours. I love that. Of course, that's how everybody wants to be invited in.

 Christina Paguyo (25:43)

Yes, absolutely invite meaning making and co-constructing knowledge together. And throughout that process build a relationship.

Allison Case (25:52)

To bring this all together. Christina, how can programs and administrators support their faculty in their efforts for inclusive assessment?

Christina Paguyo (26:02)

That's an excellent question, Allison. And so to answer this question, I invite our listeners to also be a camera lens that zooms out from the individual classroom and look at things at the programmatic level. And I would recommend that for programs that really want to conduct inclusive assessment in this capacity, that there is an audit of the curricula to see what content is covered and what authors are being represented. And do we have broad representation of content and authors? I’d also recommend a curricular map that aligns, um, of course, the required coursework with the student learning outcomes and how do, how does that align at the institutional level where there are institutional student learning outcomes that are related to diversity, equity and inclusion,

Allison Case (27:06)

Brilliant. Another dimension to add to that curriculum? I completely agree. I feel like operating without a curriculum map in any category is operating blind. That's wonderful. Well, Christina, thank you so much for your time. I’ve just so appreciated, um, just how thorough you've been with my questions. You really helped me understand, uh, or at least begin to understand and consider inclusive assessment. And I think our listeners will feel the same. I just appreciate your, your expertise and, um, the time that you've given us. So, thank you so much. 

 Christina Paguyo (27:39)

Oh my gosh. Thank you so much for inviting me to chat.

 Allison Case (27:45)

It was a pleasure. Thanks so much.

Christina Paguyo (27:46)

Thank you.

 Allison Case

Dear listener. Did you learn something new today? Maybe you’re like me and for the first time it's occurring to you that the faces smiling back at you might be hiding hunger, a death in their family due to covid, or other hardships that come our way in life. This is heavy stuff, but take heart. Like Brene Brown says, owning and resolving other people’s emotions are not our jobs.  We don’t need to feel charged to “fix” all of the struggles in our students' lives. But,  knowing of them will certainly inform our interactions with them and let them know they have a partner that is rooting for them. Christina shared the three guiding principles of inclusive assessment - viewing everyone as both a teacher and a learner, choosing a north star to guide your class, and operating from the constellation of flexibility and transparency.  

 As for the practical steps we can take, Chirstina encouraged us to start small and try some micro pivots. Christina explained that a simple stress test at the beginning of the semester, and intermittently throughout cna help us keep our fingers on the pulse of our students' whole selves. Some quick, ungraded, formative surveys will help us understand how they are progressing, where their confidence is growing, and where they could use a boost. We also heard about the importance of timely feedback to our students. Take heart, today’s technological tools, and the incredible power of computer based testing to provide tailored feedback to students at the click of a button means feedback is possible without, what can be at times, an unmanageable burden on faculty.  Especially now where student’s interactions with others are limited, sometimes our face or our voice might be the only input they are receiving.  New tools and technology can allow us to give feedback without overwhelming us.  There was so much richness in today’s chat. I hope you latched on to one thing you can try.  And I hope today opened your eyes to a whole new layer of education. 

Tune in next week to Pedagogo to hear my conversation with Brian Goldestein, Chief Academic Officer at the University of St. Augustine for Health Sciences. Brian and I talk about his path to leadership in academia, which he started on begrudgingly and which now he counts as a joy. Align the way, he shares wisdom and insight about the power of effective leadership, overcoming the stigma of being an “admin”, and the opportunities that lie ahead for universities who embrace enablement minded leadership and succession planning. You won’t want to miss. 

 Two weeks from now, Pedagogo goes out in style.  For the final episode of Pedagogo Season 2, we talk to THE Natasha Jankowski, genius, forerunner, and Executive Director of the National Institute of Learning Outcome Assessment. A philosophy major turned assessment guru, Natasha  shares her insight on the philosophy and science of assessment.  Join us as we talk about the future of equitable and enabling assessment. 

 For a transcript of today’s show, additional resources, or to listen to previous episodes, come visit us online at ExamSoft.com/Pedagogo. While you’re there, take advantage of the other assessment resources we have including on-demand webinars, white papers, blog posts and more. Until next week, be brave,work hard, be kind to yourself and others. 

 Joe Taucher

Pedagogo brought to you by ExamSoft, the assessment software that keeps security and integrity in your exams while providing you actionable data for your outcomes. For all of the toughest testing challenges, ExamSoft has you covered. 

 Keeley Karsten

This podcast was produced by Allison Case and the ExamSoft team.  Audio engineering and editing by Adam Karsten and the A2K productions crew, including me, Keeley Karsten. This podcast is intended as a public service for entertainment and educational purposes only and is not a legal interpretation nor statement of ExamSoft policy, products, or services. The views and opinions expressed by the hosts or guests of this show are their own and do not necessarily reflect the views of ExamSoft or any of its officials, nor does any appearance on this program imply an endorsement of them or any entity they represent. Additionally, reference to any specific product service or entity does not constitute an endorsement or recommendation by ExamSoft. This podcast is the property of ExamSoft Worldwide and is protected under U S and international copyright and trademark laws. No other use, including without limitation, reproduction, retransmission, or editing of this podcast may be made without the prior written permission of ExamSoft.