Putting Proctoring to the Test: A Primer on Remote Proctoring Services

Mark Spitzer is back for this bonus mini-sode to talk about remote proctoring services. If you’re already using proctoring services, Mark and Allison will discuss tips and best practices for seamless implementation of remote proctoring for your assessments in a virtual classroom. If you are new to the proctoring scene, Mark and Allison talk about the considerations you need to be aware of to make a sound choice, the types of proctoring services available and the pros and cons of each.  

Transcript

Britt: “Pedagogo” the show that brings education to your ears and meta-mastery to your assessments. Today’s episode covers all you need to know about how you can tackle student testing during the COVID crisis without breaking into a fevered sweat. “Pedagogo” brought to you by ExamSoft.  The assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Allison: Well, welcome everybody to this special mini-sode of Pedagogo. I'm your host Allison Case and today Mark Spitzer is back to talk to us about remote proctoring. If you’re already using proctoring services, Mark and I will discuss tips and best practices for seamless implementation. If you are new to the proctoring scene, Mark and I will talk about the considerations you need to be aware of to make a sound choice, the types of proctoring services available and the pros and cons of each.  

Allison: Welcome back, Mark. Thanks so much for coming back to Pedagogo. Before we talk proctoring, what’s the good word? 

Mark:  I finished Tiger King, thank goodness and now I'm onto Nailed It, the competitive cooking show where everyone is just as terrible at cooking as I am. So, definitely makes me feel good about myself which we could all use right at this point.

Allison: Well Mark, I'm glad to hear you're healthy and safe and have plenty of good TV to watch. 

Allison: So, Mark I’ve gotten a lot of questions about proctoring. There is a lot of interest as schools plan for every contingency for fall, so I’m hoping we can start with the basics.  What are the main considerations for listeners thinking about a proctoring service for the first time?

Mark: So, there's a lot to consider when you're looking at remote proctoring solutions and kind of like assessment, there's a spectrum when it comes to remote proctoring as well. On one end you have live proctors, where we would call it “eyes on glass”. You have somebody literally watching the student at all times throughout the assessment and they're able to intervene then and there. 

Mark: On the other end, you typically have some type of AI component or solely focused on AI. So, you have a review after the fact, where a computer is going to go through and flag those behavioral anomalies.

Allison:  Mark, I understand there are also hybrid models that combine AI proctoring backed up by live proctors. 

Mark:  Yeah, student is recorded and the recording is then run through an artificial intelligence engine to flag any behavioral anomalies.   A live proctor then reviews the AI flags in order to help narrow down those false positives or to provide greater context and insight in terms of the areas of true breaches of academic integrity. 

Allison: I would imagine this hybrid model plays better with students, knowing they aren’t quote unquote “being watched” while they test. 

Mark: There's been a lot about proctoring in the news and about how students feel creeped out about alive proctor watching them throughout the entire assessment, having access to their computer, and being able to see their entire room or environment.  A combined proctoring model combines these valid objections. Students do not need to give up control of their system to a live person and exam takers are not watched in real time.  Um, and so, when you're making these decisions you have to consider what's going to be most important, too.  I think that with assessment remote proctoring, no matter what you're going to be faced with a little bit of risk and it's about where you want to accept it and how much you're willing to accept. 

Allison: What about some of the issues associated with each of these methods of proctoring?

Mark: So, as an example with live proctoring. You might run into the issues where you are unable to schedule exams on your own time. You have to be dependent and reliant on a third-party organization to have the bandwidth and availability to do so. Those solutions also require persistent Internet connections. So, with students being home or streaming a lot or having their siblings or roommates or significant others in the home it can sometimes make it difficult to maintain a stable Internet connection. So, that can create challenges. Similarly, when you're relying on a fully AI engine, then in those instances you may find that you're having a lot of false positives and spending a lot of time reviewing a lot of videos unnecessarily. So, as an example I was working with an institution that had been utilizing a fully AI reviewing process. The student was testing in a hospital setting or a clinical setting so there's an EKG in the background. So, you can imagine how many hundreds if not thousands of flags were picked up within a couple of hours with the test. So, for that particular exam period the faculty member felt that it was almost worthless to them to be candid, because they had to spend so much time. So, that's why you have to make a determination based off of cost. Based off of the type of exam and then where you want to introduce and accept risk.

Allison: Mark for listeners who give assessments with a proctoring service, talk to us about building blocks to be on the path to doing so successfully.

Mark:  So, one thing I can definitely identify is to give a mock exam. You want students to be prepared and understanding of the technology. More than anything you want to make sure that you are prepared, and you've done your due diligence prior to that high stakes assessment.

Mark: So you're going to want to consider the logistics of providing that mock exam, but you're going to want to do so strategically where you can help influence and inform not only the understanding of the application but so you can start setting the precedence for things down the line.

Mark: The last thing you want students worrying about exam day is the technology. You want them focused on the content. So, I think that ensuring that you're providing a mock exam and that students are able to interface, successfully with the technology can be really key to ensuring a successful experience by utilizing that mock exam concept. 

Mark: It's also important to provide the academic integrity or exam policies at the beginning of the exam so, that way students can understand the expectations from their perspective and even more importantly sometimes acknowledge that. So, some services including ExamSoft have the ability to provide information to students before they actually enter into the high stakes environment where they are able to read, and acknowledge, notifications. So, something similar to saying that by selecting “next” you hereby acknowledge and agree to the exam integrity policies as set forth in your student hand handbook. Those sorts of mechanisms that can be utilized to ensure accountability, can be really critical so, students understand their expectations and that you can hold them accountable to it later.

Mark: I also think that it's really important to communicate a timeline and expectations to students, where you are able to mitigate any of their concerns. Also, that you set expectations with them ahead of time, so you understand how long it's going to take to get some of those reports back from the proctoring organizations. 

Mark: So, you can set the expectation of how long you're going to wait before you release results back to students. So, this is going to help ensure that they're anxiety isn't through the roof and that you're setting clear expectations and guidelines internally. The next thing I would mention is that it's really important to understand the type of organization that you're partnering with. So, understanding what the company offers, ensuring that you understand exactly how they prevent the student from cheating or mitigate concerns related to cheating and what happens in the event that they were not able to identify certain trends or they're not able to identify specific behavior. So, it's about understanding the limitations of the system. It's about understanding what you can do to be proactive in communication. And it comes down to ensuring preparedness by delivering that mock exam. 

Mark: You're also, going to want to ensure that students are able to confirm the minimum system requirements including the webcam and the microphone. 

Mark: It's important to ensure that you have a plan in place, prior to exam day on how you want to help ensure that any exam day challenges that arise, are dealt with quickly and efficiently. So, you're going to want to review the plan, make sure that you have input from appropriate stakeholders, reach out to the appropriate individuals at your vendor partner to ensure that you're all aligned on the strategies to ensure that exam day is as smooth as possible. That really again goes back to finding the right tool and the right company to work with. 

Allison: Mark, let me jump in there. Help us by rattling off some of the questions we might ask ourselves and a vendor partner as we consider remote proctoring solutions. 

Mark: Do you need that live proctoring? Do you need something that is AI? Do you want the best of both worlds?  And how quickly can they on board you? How well can they support you and you want to focus on finding a partner company that's going to help you grow and achieve your goals. Not only to overcome the immediate urgency of the current crisis related to COVID but looking long-term to ensure that we're providing the best educational experience possible to students and faculty alike.

Allison: Mark, for people who are listening and who are realizing, they really do need this proctor solution, is it feasible to onboard proctoring solutions over the summer?

Mark: So, it depends Allison. It depends on the type of service you're going to be utilizing and the bandwidth, that those vendors might have. So, what I've heard through the grapevine is that there are certain live proctoring organizations, that at the moment are struggling to keep up with demand. Which is understandable, right? I mean this is unprecedented time and we've seen a huge increase in volume. But for live proctoring, specifically, you might find that it's a bit more challenging to on board quickly. 

Mark: Because you're going to have to partner with them in terms of determining schedule and availability of their team. Which may be a bit more limited at this time. If it's something that's going to be less reliant on live proctors and it's something where they can be little bit more flexible in the technology, then you're going to find that it's going to be a much more seamless process, for on boarding and that the bandwidth is going to be much more attainable because of the way that the different technology operates and the way that onboarding can be accomplished with those variances.

Allison: This is really fantastic information, Mark because it looks like proctoring might serve a critical function in fall, and it’s important that listeners know their options. From eyes on glass to AI and everything in between, each type of service has its advantages and disadvantages, and knowing those is so important, but then  beyond the technology itself, considerations like “how long it takes to get the proctoring report back from the proctoring service so I can release results to my anxious students who want to know how they did?” That wouldn’t have been on my radar before. And the availability of the proctoring service to work on your timeline are important considerations like that, the intangibles, that aren’t immediately obvious to those who are new to the proctoring world. 

Allison: To wrap us up today, Mark, will you recap the tips and building blocks for remote proctoring solutions?

Mark: We talked about several building blocks including considering logistics of providing a new mock exam where students have the opportunity to test out the technology firsthand. Establishing academic integrity and exam policies to help influence the direction of the assessment to ensure that students are confronted with that information before they enter into the assessment itself. Having a plan. Ensuring that you understand what you can do to help mitigate concerns on exam days and what resources you have available in the event that you do need to reach out for additional help. And lastly, it's about finding the right tool and the right company to ensure that you are as successful as you can be, come exam day.

Allison: Thanks for that, Mark. Listeners, we really hope we gave you best practices that will lead you to success in administering your proctored assessments, or if you’re in the market for a proctoring solution, a quick lay-of-the-land to help you navigate the choice that’s right for your program. Mark thanks so much for joining me today. I can't wait to hear what you've tried to bake the next time we talk.

Mark: Nailed It.

Britt: “Pedagogo” brought to you by ExamSoft. The assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Keeley Karsten: This podcast was produced by Allison Case, Mark Spitzer and the ExamSoft team, audio engineering and editing by Adam Karsten and the A2K productions crew including me, Keeley Karsten. This podcast is intended as a public service for entertainment and educational purposes only and is not a legal interpretation nor statement of ExamSoft policy products or services. The views and opinions expressed by the hosts or guests of this show are their own and do not necessarily reflect the views of ExamSoft or any of its officials, nor does any appearance on this program imply an endorsement of them or any entity they represent. Additionally, reference to any specific product, service, or entity does not constitute an endorsement or recommendation by ExamSoft. This podcast is the property of ExamSoft worldwide and is protected under U S and international copyright and trademark laws. No other use, including without limitation reproduction, retransmission or editing of this podcast may be made without the prior written permission of ExamSoft.