Pedagogy, Not Panic-Gogy Part 2

Allison and Mark talk about the benefits of remote assessment, the whys and why nots of throwing in the towel and just assessing students using “group chat” for the remainder of the semester, and some common stumbling blocks and building blocks when attempting to execute fair, high-fidelity, secure remote assessments in a COVID-environment. 

Transcript

Britt:   0:00
“Pedagogo”, the show that brings education to your ears and meta-mastery to your assessments. Today’s episode covers all you need to know about how you can tackle student testing during the COVID crisis without breaking into a fever and sweat. “Pedagogo”, brought to you by ExamSoft - the assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Allison:   0:31
Hey Education Nation, welcome to Part 2 of Pedagogy not Panic-Gogy.  I'm Allison Case, a former educator and Director of Curriculum and Accreditation at one of the largest biomedical engineering programs in the nation and now an education consultant ExamSoft and podcast host.  And I'm here with my guest cohost this week Mark Spitzer, Associate Director of Client Solutions at ExamSoft.  Mark, thanks so much for joining me again for Part 2 of Pedagogy not Panic-Gogy.

Mark:   0:58
Thanks for having me, Allison.

Allison:   1:00
So it's week  #111 in quarantine,  Mark. Do you have any quarantine related news for us?

Mark:   1:05
Oh yeah, so on Twitter there is an account now where it's just dedicated to rating people Zoom set up.  So, they're looking at their living room and their art choices and what kind of books do they have?  Their own book?  Did they write their book?  How did they strategically place it so that you can read it? And they're going in and giving everybody scores.  And the crazy thing is that people are actually responding and making decorating choices off of what this random Twitter account is saying, so gosh, I know that social media is important and pretty influential, but to impact the way that you position your books or hang your artwork? That's, uh, something else and definitely something that's only possible in the age of COVID.

Allison:   1:40
So, Mark, I think what you're telling me is someone's job is to decide what books and backdrop you will have before you start your Zoom call or your live feed.

Mark:   1:49
Hey again, it’s a good thing that we're doing a podcast because having somebody go through all of that trouble would be a full-time job, I think.

Allison:   1:56
Last episode we talked about 3 features to focus on and preserving this transition to a virtual classroom: online teaching which is anytime you're sharing new content with your students, the value of community and building that community in all of your interactions, and today we continue the conversation by talking about how to continue with assessments in a post-COVID virtual classroom.

Mark:   2:16
Allison, how does assessment factor into all of this?

Allison:   2:19
Great question, Mark. Assessment tells us what our students do and do not know. And beyond being able to just give them a grade, that information, that feedback is critical. It allows instructors to scaffold what's coming next and upcoming content onto current student understanding. And really effective instructors are adept at being able to relate what is new to what is known. Next assessment gives students information. One, it allows them to more clearly understand what they do and do not know. And two it allows for that process of metacognition. Studying and understanding your own thinking.

Mark:   2:59
I can definitely see how important self reflection would be in that process.

Allison:   3:03
So, assessment is critical to the fiber of our teaching experience, whether it's face to face or in a virtual environment. So, as we transition to a virtual environment, just asking ourselves “what roles can low and high-stake assessment play?” And where can both summative and formative assessment be used to again inform faculty and provide students the opportunity to demonstrate and understand what they do and do not know.

Mark:   3:30
Allison, I absolutely agree that assessment continues to be critical. Not only are you going to be able to keep students engaged and motivated, but you're also going to be able to provide them with timely feedback. I think that providing them with that feedback at this point in time is of the utmost importance because it helps keep their eyes on the ball. As we start moving into the summer months and students start thinking about their post-COVID plans, it’s important to ensure that we're maximizing this time and providing timely an informative feedback via reports and analytics.

Allison:   4:02
So, Mark with all of the stresses and all of the changes our students are facing in the past few weeks. Why stick with assessment? We have tests from the first half of the semester and why not just make the rest of the semester “Group Chat”?

Mark:   5:12
Yeah that’s a good question. I'm sure there's a lot of students out there that would hope that I would say, “yeah let's just forget assessment”, you know? We've all been through enough at this point. But it really still is key. It's really critical for people to continue assessment. I think one of the biggest factors is that it helps motivate performance. You know? It's easy for students to slide back or to become demotivated throughout this experience. They’re not going to class; they’re not meeting with friends. But what's really important is that they are continuously motivated.  And assessment is really a key driver of that. To ensure that they have goals, that they have milestones, and that they're receiving timely feedback about their progress. This is especially true if you're utilizing some of the category recording thing that we have ExamSoft. So, you can help them understand some of their strengths and weaknesses or strengths and opportunities. To ensure that they're really focusing on the things that they need to and making the most of their time at home.  

Allison:   5:12
What do you mean by strengths and opportunities?

Mark:   5:14
All students have topics that come easily to them, or that they really understand, and those students have other topics where there were some weakness, or as we like to refer to it, an opportunity.  By informing both students and instructors alike on the strengths of each student as well as the areas of opportunity for each student, both instructors and students will be better informed. For instructors, they will be able to better understand how they teach and where the cohort might be struggling or excelling. Students will be informed on the specific areas in which they need to study, where they need to focus their time and what even might be worth going to office hours for. And again this drives metacognition as you were talking about earlier, Allison, if the students can identify “Hey, I'm great at every single topic on this assessment, as long as it's presented to me as a recall question or an understand question”,  then they can have that light bulb moment of “oh it's not the content that's holding me up it's the application of the material in a real world setting”. Insights like those happen every day when students and instructors are given data about performance and not just an exam score.   Strengths and Opportunities reports and those like them also give faculty insight into what no student understands.

Allison:   6:21
Listeners with digital assessment platforms, this kind of information would be found in the reporting and analytics section, but  for those of our listeners without digital assessment platforms, let me give you some tips here on how you could develop your own time for reflection and identification of strengths and opportunities.  First, I encourage all instructors to do an exam review where they share the class average the range of scores and the numbers of A, B, C,D and Fs. I did this every time I would give a high stakes assessment and this really level-sets students in terms of how hard they felt the assessment was versus how their peers performed.

Mark:   6:58
I know sometimes folks are a bit hesitant to share that data

Allison:   0:00
Mm-hmm

Mark:   0:00
But I think transparency is key agree with you Allison.

Allison:   7:04
Next, it's worth instructors using class time to review the top one or two most missed questions.  Instructors can make this interactive asking students to raise their hands or, you know, press their ‘raise the hand button’ on Zoom, and give the next right step and this allows students to process the procedure or the correct solution with their peers as well as hear the material from their peers.  Finally, I would encourage all of our listeners to give students maybe even just five minutes in class to categorize the questions they got wrong on an assessment.  That will really direct their studying.  And instead of having to review all of the material in the assessment, give them the opportunity to really focus on the topics that they performed the lowest on. Mark, keep talking to us about assessment and why it's worth the effort to continue assessing in this new landscape of COVID.

Mark:   7:55
The other thing that it really helps do is ensure that educators are setting standards appropriately. I think this is true for individual faculty but also pragmatically or at the administration level, as well.  It's easy for it to feel like the Wild West right now.

Allison:   8:00
Mm-hmm

Mark:   8:01
I think that one thing that's going to be really critical is for different departments and different teams to align themselves on these standards. What better way to verify that we are actually assessing the things that we need to, than by using assessment in all of the shapes forms and sizes.

Allison:   8:25
You bring up a good point.  When I'm giving an on-site workshop and working with programs, I consistently advise programs to have faculty meetings dedicated to standard setting and benchmarks that the entire faculty know, understand, and agree upon because without this cohesion, programs are really missing opportunities to send consistent messages to students about performance and their readiness to practice. A quick example would be “what grade on a high stakes assessment warrants mandatory office hours and some sort of remediation?” And instilling this consistency is a simple as putting time on the schedule to have the discussion it's such a simple step but it can have a big impact on students.

Mark:   9:04
So, I think you made a really great point when it came to like “formative” and the “summative” assessments or these high stakes versus low stakes assessment in the last episode. The critical thing is that we continue assessing. The critical thing is that we are ensuring that we're providing an equitable experience, even if they're not in the classroom, because at the end of the day, it's going to ensure not only that they're successful, but as a program and as an institution you're continuing to set the bar high. Instilling that same level of expectation in the students themselves.

Allison:   9:30
Mark, you touched on so many of the direct and indirect benefits of assessment. You're absolutely right. Maintaining motivation as we shelter in place and as life changes so drastically, is tough.  Assessment will play an important role in demonstrating how effectively our students have transitioned to learning in this virtual environment. And really serving as a check in. Mark, help us with some of the stumbling blocks and building blocks as we seek to reintroduce assessment into our virtual classrooms.

Mark:   10:00
The first is establishing exam day best practices. You're going to have to rethink the way that you do assessment when you're moving from in-person to like this virtual classroom. An example of that is how long your exams are. I've seen exams range anywhere from an hour to three hours plus. One of the challenges that presents itself with remote testing is what do you do about simple things that we all took for granted, like restroom breaks, as an example? Obviously if you're recording the student doing the exam, when they get up and they leave the computer to go to the restroom, well it's like the Wild West again! You have no idea where that students going, what resources they might have available to them stored in the other room. So, instead of introducing that risk, you need to rethink your exam day policies. Maybe we're going to start trunking these exams. Maybe they’re going to be an hour, maybe 90 minutes in length, max, and then we will give them additional exams throughout the day so that way we can help hold them accountable and ensure that they're not leaving their seat prematurely. You also want to find something that helps mitigate risk. I am not only talking about from the institutional perspective but from the student perspective too. As scared and apprehensive as we might be in the education community, I think that the students are also feeling that same level of anxiety. “I'm going to be taking my high-stake final exam online or on a computer, what does that mean for me? What happens if my network goes out? What happens if it's raining outside or my power goes out?” These are all the variables that students think of and so providing a tool that's going to help them feel confident on exam day, I think is really critical. So, um, obviously, with ExamSoft we can allow for testing to be done offline and securely. So, if somebody, their roommate or their neighbors are hogging all the bandwidth, you know watching 4K videos all day, what they can be assured of is that there's testing experience is not going to be impacted by the behaviors of others. We all want to have parity; everyone wants to be on an equal playing field. So, if some students are buffering the others aren't, that's obviously a poor experience, so, find something that's going to give everybody that equitable testing experience.

Allison:   11:34
Mark it’s good to know your roommates “Tiger King” binge watching won't affect your exam turn in.

Mark:   11:35
Yeah as long as they don't get too far ahead because I have some episodes I need to catch up on.

Allison:   12:05
I'm so glad you brought this up I'm familiar with lockdown browsers because anytime I gave a test in the testing center with an on line portion I had the option to use a lock down browser but what is a lockdown browser versus an application that can lock down the entirety of the device and what is each design to do?  And as we look for COVID solutions how can each of these tools help us out?

Mark:   12:11
It's a great question, Allison, and there's definitely a distinct difference between the locked down browser and a lockdown application designed to secure the entirety of the device.  First let's define what a lockdown browser is. In essence, a locked down browser might be used to prevent students from accessing any other websites and using basic functionality on the device like copy and paste.

Allison:   12:14
That sounds great to me and it seems to cover all the bases. Why then do applications to lock down the entire system exist?

Mark:   0:00
Well, you’d certainly think so, but there's more to consider.  If you Google “how to bypass a locked down browser” you'll get a ton of results.  So, security is a spectrum and you'll have to see where your comfort level lies.  When giving high stakes exams mid-COVID, I'm hearing more and more educators share concerns that services they've used in the past just no longer cut the mustard.  And so, it's clear that the first major difference between lockdown browsers and dedicated lockdown application is security.  Within Examplify, as an example, which is the testing application that locks down the entirety of the device here at ExamSoft, it ensures that there are no third-party applications like “WhatsApp” or web browsers or Spotify from being utilized.  It also prevents you from using ALT-TAB to switch between windows, which is something I found with a two second Google search that is possible with lockdown browsers.  In short, an application that can lock down the entirety of the device is like a sledgehammer designed to mitigate cheating.  The next big differences exam stability with something that can lock down the entirety of the operating system, or at least that's the way that exemplify works, we’re able to test offline.  So, because of that, you're not going to have any change in experience from the student perspective, because of a degradation of service or lost data because there's an Internet connection that goes out.  So, it's really key and critical to ensure stability on exam day and it's been our experience that by utilizing something that can operate offline you're going to create a more seamless exam experience.  

Allison:   14:25
So, applications that lock down the entirety of the device or typically packaged in with software designed to give high stakes assessments?

Mark:   0:00
In comparison, most locked down browsers are used in conjunction with other products like an LMS, which are really designed to communicate with students and not truly designed for delivering high stakes exams.  The purpose is to give security to an online based assessment. But lockdown browsers will not prevent students from using textbook or pulling out a cell phone, so additional proctoring services must be bought from other companies and should be used for appropriate. Last, but definitely not least, is that you want to find an assessment solution that's going to be robust. You want to find variances in question types that's going to allow for you to grow and expand outside of what you might be typically doing here and today. You also want to be able to find new and exciting ways to use assessment outside of your typical “MCQ” exams you might be utilizing today. I love “MCQ” exams but in these times where we have a lot of distance education, we also need to ensure that we're able to find a solution where we can test in these different ways that we've been talking about – in class,  formative,  obviously high stakes and proctored. So, finding something that's going to allow for you to build that bridge between all of those assessment types is going to give you the information that you're going to need to start painting that picture of the student experience.

Allison:   0:00
Mm-hmm  

Mark:   0:00
It's also going to be really important for you to know at the end of the day, when you start looking back at this whole experience when we're months or years separated from all of the craziness that we're living in right now, to know that you were addressing an immediate need, yeah sure, but you’re also building for the future. Um, and I think that this is an opportunity, as difficult sometimes as it is to think about this as an opportunity, but it is an opportunity for us to rethink the way that we're doing assessment,  it's an opportunity for us to rethink how we can engage students outside of just the classroom. And it's an opportunity for us to rethink how we can utilize not only a secure testing application and platform, but also the data to better inform our student’s experience. So, I think that all of those three things are going to be really critical, when you're considering the building blocks of ensuring strong assessment.

Allison:   16:42
I completely agree with you. It’s a unique occurrence to be forced to rethink what you’ve been doing for years, and what better time to make informed decisions about assessment tools and practices that can pay dividends well into the future?  I know for my own personal COVID experience, I’m trying to remind myself to see the good every day and identify the unique opportunities in front of me. What are some of the stumbling blocks when it comes to re-introducing assessment in a virtual classroom?

Mark:   17:10
One of the biggest stumbling blocks, is just not using a tool that's really designed for remote secure online assessment. I know that a lot of people are looking for solutions that are going to address their most immediate need. What you want to try to avoid to do though is finding a band-aid solution where the long-term impact is actually a net negative for you.

Allison:   17:32
I'm guessing here Mark you're referring to things like trying to Zoom proctor an assessment and other work arounds in the absence of a secure remote online assessment tool.

Mark:   18:53
Yeah, I think there's definite value and utilizing teleconferencing software like Zoom to keep eyes on students, but it's not really a long term or scalable solution.  When you think about the time and energy that's going to have to go into the proctoring of students, or to review each and every one of their exam sessions. It becomes a bit burdensome, and so looking for tools where you can feel confident and comfortable in the diligence of the process but also knowing that you aren't going to be impacting your own, or your team’s, bandwidth can be really critical when you're thinking long term. Um,  I think that if you start doing things the right way, with the right intention, with the right attitude and if you’re doing it in a certain time frame here, which is obviously expedited, but knowing that you're building towards something bigger, I think will help prove, long-term, that you were doing things the right way the first time. One of the other stumbling blocks I tend to see is just not evolving quickly enough. Especially when we're talking about ways to mitigate academic dishonesty during remote assessment. So, you know I think that it's easy for us to kind of use the same features and different options that we've done in the past hoping that it works out.

Allison:   18:55
But as they say, hope is not a strategy.

Mark:   0:00
Right. So, imagine you're giving a test remotely and student home is in a virtual conference with their classmates talking about your assessment. Let's figure out some ways in which we can utilize tools that we have our disposal to help mitigate some of these challenges.

Allison:   0:00
I was talking to a colleague last week, Mark, who her primary concern is that students are at home on a Zoom call taking her assessment, so I couldn't agree with you more.  

Mark:   0:00
So, maybe it's like randomizing your question sequence or randomizing your answer orders. I also wanted to highlight a couple of other things that not everyone has been thinking about. So, putting time pressure on students is really important. If you have a time crunch, students are going to be a lot less likely to try to call or send a “WhatsApp” or whatever the case might be to try to determine what’s on the exam.  If you’re not using some of those remote proctoring solutions, then having really tight time limit is going to be critical to mitigate some of these concerns. Obviously if they're using proctoring, that's not going to be an option to them because they’re going to be busted if they try to cheat in that manner. It can also be a really great to prohibit backwards navigation. So, especially if you're having unfolding case studies, you're presenting additional information as you progress throughout the exam, it can be helpful to prevent them from going backwards in the exam, but specifically, when you're trying to mitigate it for your distance learners, I think that it's really key to help set guidelines and expectations for the student that they need to be able to understand the sense of urgency and that time pressure and you can also help provide that by preventing backwards navigation. So, you want to be able to lean on some of those best practices as well as some of the remote invigilation tools to ensure that you're providing the most secure testing experience possible.So, maybe it's like randomizing your question sequence or randomizing your answer orders. I also wanted to highlight a couple of other things that not everyone has been thinking about. So, putting time pressure on students is really important. If you have a time crunch, students are going to be a lot less likely to try to call or send a “WhatsApp” or whatever the case might be to try to determine what’s on the exam.  If you’re not using some of those remote proctoring solutions, then having really tight time limit is going to be critical to mitigate some of these concerns. Obviously if they're using proctoring, that's not going to be an option to them because they’re going to be busted if they try to cheat in that manner. It can also be a really great to prohibit backwards navigation. So, especially if you're having unfolding case studies, you're presenting additional information as you progress throughout the exam, it can be helpful to prevent them from going backwards in the exam, but specifically, when you're trying to mitigate it for your distance learners, I think that it's really key to help set guidelines and expectations for the student that they need to be able to understand the sense of urgency and that time pressure and you can also help provide that by preventing backwards navigation. So, you want to be able to lean on some of those best practices as well as some of the remote invigilation tools to ensure that you're providing the most secure testing experience possible.

Allison:   20:41
Well and Mark, I just love hearing just by using current tools and even current settings that faculty have available to them, faculty have the power to influence their student’s test taking experience and mitigate some of that cheating.

Mark:   0:00
Mm-hmm. I've been talking to a lot of folks recently and one of the stumbling blocks, that I tend to see is a rigidity when it comes to determining the style of assessment. So, you have to remember that not all assessments are created equal. In some instances, it makes sense to have that high stakes, secured testing, where you have them proctored. You're going to be utilizing “Exam ID” and “Exam Monitor” software to ensure that you're installing the best exam integrity policy possible. Sometimes though, you know, you really need to have some of those open book exams. Maybe have constructed responses, you're giving them the opportunity utilize tools or resources like the internet in order to write compositions or essay responses that are done over multiple days. This is a great work around if you don’t have a secure digital assessment solution or remote proctoring tools, because it still queries what students do and do not know but it does it in a way that requires higher order thinking and an opportunity to show what the student knows in a way that can be Googled. The take home messages of valid assessment can be conducted in a lot of different ways.

Allison:   21:57
I agree there is no one right kind of assessment, and no one size fits all.

Mark:   22:02
The more robust assessments you have, the more data you generate, the more data you have, the better you're going to be able to make those data driven decisions.

Allison:   22:10
Mark, talk to me about how you've seen faculty most effectively use data to inform their teaching.

Mark:   22:15
The most effective faculty I've spoken to are assessing all of the time.  And this isn't as cumbersome as it seems - starting lecture with a question from the last class, asking for a show of hands, using an audience response tool - all of these formative assessments are data points.  The most effective teachers I talk to are assessing every time they are in class whether it's formative, low stakes assessments, or summative high stakes assessments and they generate enough data points to one, established patterns and two, gain insights on how to proceed.

Allison:   22:43
You know when I was teaching, I would conduct at least two formative assessments per class. I know they say that students can listen between 12 and 18 minutes of lecture at a time before tuning out.  And I know you know, even my own tendency, is when I'm attending a lecture, unless you're taking steps to be engaged in active listening, it's easy to tune out and go into notetaking mode and not really engage your mind in what you're hearing, and so these types of on-the-spot formative assessments really do impact student retention and instructor understanding. But to your point,  the more data points, the reason I say all that is, the more data points you have about student performance, and the more diverse your data points are, so, formative and summative, high stakes and low stakes, oral and written, project based, didactic coursework, the more insight you're going to get, and it doesn't have to be done in a way that students feel like they're being you know quote unquote “assessed”.

Mark:   23:41
Absolutely, but I would just recommend not falling into a particular routine. When it comes to assessing the same way over and over again. Mix it up a little bit. Have a little variety in the way that you're doing assessment. Even if it's a little outside of your comfort zone or something you haven't done in the past.

Allison:   0:00
To wrap up today, let’s recap the building blocks and tips we discussed for continuing with assessment in a virtual classroom.  We discussed the value of continued high-quality assessment. The building blocks of establishing exam day best practices, using settings and testing conditions to mitigate academic dishonesty, like question randomization, answer option randomization, time constraints and preventing backwards navigation.  And choosing a tool to give students confidence and parity in their testing experience.

Allison:   24:55
For more information you can see details about some of the topics we talked about today in our show notes. If you liked us, be sure to share with your colleagues. And be sure to subscribe so that you can be made aware of our next episode. In episode three will be talking about what to look for in a digital assessment solution. My thanks so much to Mark Spitzer for joining us today. Mark, thank you for coming and providing just such wonderful answers and insights into what your clients are facing amidst these very strange new times.

Mark:   24:57
Thanks so much for having me also and I really do appreciate it.

Allison:   0:00
 Thank you. Stay safe and stay well.

Britt:   0:00
“Pedagogo” brought to you by ExamSoft, the assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Keeley:   0:00
This podcast was produced by Allison Case, Mark Spitzer and the ExamSoft team.  Audio engineering and editing by Adam Karsten and the A2K productions crew including me, Keeley Karsten.  This podcast is intended as a public service for entertainment and educational purposes only and is not a legal interpretation nor statement of ExamSoft policy, products, or services.  The views and opinions expressed by the hosts or guests of this show are their own and do not necessarily reflect the views of ExamSoft or any of its officials, nor does any appearance on this program imply an endorsement of them or any entity they represent.  Additionally, reference to any specific product, service, or entity does not constitute an endorsement or recommendation by ExamSoft.  This podcast is the property of ExamSoft Worldwide Inc and is protected under US and International Copyright and Trademark laws.  No other use including, without limitation, reproduction, retransmission, or editing of this podcast may be made without the prior written permission of ExamSoft.