Pedagogo

Better Than a Ticonderoga #2: What to Look for in a Digital Assessment Platform

May 05, 2020 Season 1 Episode 3
Pedagogo
Better Than a Ticonderoga #2: What to Look for in a Digital Assessment Platform
Show Notes Transcript

Allison and Mark talk about what features to look for in a digital assessment platform.  We discuss the "need to haves" and the "nice to haves" including syncing with an LMS, seamlessly integrating with proctoring services, supporting new and emerging item types like hot spots, easing faculty burden, and generating data in support of accreditation efforts. 

Britt:   0:00
Pedagogo, the show that brings education to your ears and meta-mastery to your assessments. Today’s episode covers all of the items a digital assessment platform needs to pass the test. Pedagogo, brought to you by ExamSoft the assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Allison:   0:00
Welcome to episode three of Pedagogo. Once again, I'm joined by Mark Spitzer. Mark, thanks so much for joining me again today.

Mark:   0:00
Thanks for having me, Allison..

Allison:   0:00
So, just a heads up to our listeners, be sure to join me starting next week as I interview faculty members, provosts and directors of curriculum to talk about how different programs are preparing for fall and how diverse programs are defining success in a post-COVID fall term. So, Mark today you and I are going to talk about what to look for in a digital assessment solution. I think this is going to be a really important conversation for our listeners because think because many suddenly find themselves looking for a digital assessment solution to use in their virtual classrooms. So, Mark to kick us off, what is a digital assessment platform?

Mark:   0:00
Yeah, it’s a really great question, and I think that for a lot of people can mean a lot of different things but the way that I interpret it is a one-stop-shop, someplace where you can do item banking, item authoring, exam authoring, exam delivery, and then reports and analytics.  So, it's generally going to be something that's going to be really focused on assessment, giving you all the bells and whistles that you might not necessarily see through other services that have an assessment component, but maybe not so much focused on assessment itself.

Allison:   0:00
What is the real difference between a dedicated assessment tool and testing through the LMS?  

Mark:   0:00
A lot of times you'll see that people are utilizing the testing that's available within the LMS, which is great in some respects, but when you start looking at a true, dedicated assessment platform, you’ll start to notice that some of those differences are pretty glaring and important in terms of providing strong assessment. LMSes are really designed to communicate with students and not truly designed for delivering high stakes exams. They lack in features like powerful adjusted scoring options, subjective and objective grading formats for 360 view of student learning, and item level categories which is great for reporting in-organization.  Furthermore, they are browser based, which can be a disadvantage for students with the unreliable Wi-Fi, and they can also be problematic for assessments that utilize a lot of multimedia files.  In addition, LMS testing can be non-secure, they are used in conjunction with additional products and software to make assessments secure like a locked down browser or third-party vendor tools, but LMSes, in and of themselves, are generally not the most secure way to deliver exams.  And when the exam is complete, LMSes can report the score, but they lack in some of those backend analytics that you might be looking for.

Allison:   0:00
So, what are some of the things to look for, if I'm considering a digital assessment platform?  What are the “Nice to Haves” and the “Need to Haves”?

Mark:   0:00
The nice to haves are settings for faculty and student ease, administrative conveniences, and robust and diverse item types, including new items types and those that support video and audio files.

Allison:   0:00
Great point, Mark. Can you talk to me a little bit more about that?

Mark:   0:00
One of the things that I think is really important is the ease of use, that's not only for the end user but also in terms of ramp up, and implementation, adoption, you know how can you make the lives of your students your colleagues, as well as IT team or department, make everyone's lives a little bit easier?  So, one of the things that you should look for is the ability to do a lot of auto grading, especially those objective question types, with the ability to do adjusted scoring in a lot of different ways. So, I think that one of the struggles that a lot of people have, with pen and paper testing especially, is how they manage adjusted scoring.  With ExamSoft, at least, it's great because we have the ability to allow for adjusting of the scoring metrics easily and comprehensively for the entirety of the exam with a single click.  So, ensuring that you're improving efficiencies by adding new technology is really critical.  I think everyone in general, you know, we're so inundated with different technology and software that we're really not looking at something new unless it's going to actually make our lives easier, and I think that that should be true of your assessment platform as well.  

Allison:   0:00
OK. So, faculty ease and time savings. I'm definitely with you there. What features will get faculty the ease and time savings?

Mark:   0:00
A few quick examples: not having to make multiple exam versions because now you can now randomize question sequences and answer choice. Also, the ability to create multiple question versions, and track edits to questions overtime. The ability to build exams using pre-templated exam blueprints, knowing how long exams are going to take before you administer them. And for students, providing feedback back to the instructor, utilizing notes and feedback, or allowing multiple highlighting options available, and of course the calculator can be really helpful. Another way that I've seen that ease of use come into play especially when it comes to just the everyday management of the system is if there's integrations available. Typically, when we're talking about integrations were doing so in two different ways.  The first of which are with LMS systems so with ExamSoft as an example we have the ability to pull in all of the course and roster information from four of the major LMS systems: Canvas, Blackboard, D2L, and Moodle and then to push final exam grades, after you've done your adjusted scoring, like I just mentioned, back to the LMS gradebook. So again, improving efficiency, so giving your team and your faculty more tools, is really critical.

Allison:   0:00
Mm-hmm

Mark:   0:00
So, you put more time on people's calendars to do the things that are going to make the biggest impact on their lives and the lives of their students. The other integration we tend to see is single sign-on, so things like LDAP and SAML, which I might be getting really into the technical weeds here, but really single sign-on, is determined, is designed to help students and faculty utilized the same username and password across all of the different services on campus. So, it just means that it's an easier login process for your students and your staff -  it's one less password that you have to remember, which I think that we could all benefit from, given how many times I have to reset my password on a weekly basis because I just can't remember it.  I also think that you want to look for an assessment management system that has a robustness when it comes to the functionality. You want to look for something where it has a lot of capabilities, even if you're not planning on using today, that you can start to grow into. So, I worked with a lot of programs that came on board the ExamSoft family and we're really focused on MCQs, as an example.

Allison:   0:00
Mm-hmm

Mark:   0:00
Everything was multiple choice and they knew that that was going to be what they were planning on doing, at least for the first year with ExamSoft, but knowing that they had a lot more flexibility in the item types for assessment types gave them a lot of unique confidence and how they wanted to grow their program. So, using hot spot items, as an example, allowed for continuing usage or increased usage of multimedia that's just not possible on pen and paper.  

Allison:   0:00
How does that translate to student and faculty experience?

Mark:   0:00
From the student experience, the venue is going to, of course, be changed and the technology is going to be changed. The actual assessment itself, really shouldn’t change or should improve actually, especially if we are using the great question types and multimedia so they may find that there’s value. Plus, if they’ve never gotten any feedback or reports before, and now all of a sudden, you’re giving them tremendous, detailed analytics, and specific targeted mediation techniques, boy, you know, from that perspective, I think they're going to be really thrilled. From the faculty perspective, things are definitely going to change. You have to think about how you can utilize technology to accomplish some of the things that you could do in person before.  It's about utilizing the tools you have, evaluating the marketplace, to see where there are deficiencies in your current process, and then relying on some of the different and innovative ways that you can utilize technology to assist with providing a comprehensive experience.

Allison:   0:00
Ok, so listeners will look for platforms with nice to haves including features that ease faculty and student burden, with administrative conveniences like LMS integration and single sign on, and robustness, meaning a platform with question types, feature and functionalities that a program or university can grow into over time, instead of growing out of quickly. Now, what are the must haves if a listener is going to appropriate funds for a digital assessment platform to propel their assessment into the 21st century?

Mark:   0:00
You know, security is really critical because in order to make strong data driven decisions, you need to have it exam integrity.  If you have dirty data in, you’re going to get dirty data out, so you need to ensure that you have a robust and secure testing platform. Once you have the ability to make strong data driven decisions it's going to help influence the ways that you can improve efficiencies from the faculty, and it's also going to expedite your ability to complete any accreditation review, as well, especially if you're using categories and different data points to help show that you were teaching in the way that you expected to teach, and in the volume at which you expected to teach on certain learning outcomes. Oh, you need to ensure that you have security wherever and however you can find it

Allison:   0:00
Mm-hmm

Mark:   0:00
Whether that's through the secure testing application, whether that's utilizing best practices,  mitigating concerns by randomizing question sequences and answer choices,  or using remote proctoring, all of those things together can ensure that you have that security that's going to be really important and influences everything else that you do in the system.

Allison:   0:00
Talk to me about what you mean when you say, “dirty data”.

Mark:   0:00
Yeah, so dirty data” is a term that I'm referencing when it comes to how we can ensure that we're actually providing reliable and valid information. So, when you're making a data driven decisions, you're doing so with evidence. But if your evidence is murky, because a lot of students are cheating, or content is out there, then you're making your decisions based off of information that is false. It's really important to ensure that when you're looking at information related to statistical analysis, when you’re making “data driven” decisions, that you’re doing it with the understanding and the knowledge that the data was collected in a way that is defendable, where it's replicable, and where you can point to the stability and the reliability of the system.  So, you know, that’s where best practices come into play. To verify that the student is, “who they say they are” and that they're taking it without outside help. If you try to make those big picture decisions, without having “clear data” or “accurate data”, then you may be tweaking things in the wrong direction. 

Allison:   0:00
Security is such a complex topic. You're talking to me about the security of my own items, as well as securing the test itself so, students won't have access to outside information during the exam.

Mark:   0:00
I think it's really critical to think about security not only from the IT perspective, but also what it means for you as an administrator or faculty user, and for your students, as well. We're talking about students not being able to use their device to look up the answer to the question, either on the internet, or a saved resource that they might have on the computer. This could also include not being able to override some of the proctoring functionalities, by either turning off the webcam or turning off the computer, or having a buddy turn off the internet. If you were utilizing a service that requires a persistent internet connection, there's always that inherent risk that, either intentionally or unintentionally, that there's going to be a disconnection from that live individual. When we're talking about security, we're talking about ensuring that not only are we are providing you with the tools and the resources necessary to ensure that secure environment, but also, that we're thinking two steps ahead of the students. We feel that the majority of the security vulnerabilities that might exist, and when it comes to testing exist specifically, because of a reliance on the network and a reliance on a browser that that that doesn't have the same level of attention to security as what the Examplify platform might have.

Allison:   0:00
Mm-hmm. I know I spent a lot of time and energy ensuring the security of my assessments, while the assessment was in progress. Walking the room, seating students every other seat. But I spent just as much time, if not more, writing and cultivating my favorite test items, year after year. Talk to me a little bit about the security of my items from being released or compromised.

Allison:   0:00
We know how much time and effort goes into writing those items. The last thing you want is for them to show up on “Quizlet” or on “WhatsApp”, or other ways that students communicate. I think that that's a real, valid concern. You want to make sure that students aren't going to be able to use screen capture technology or take a screenshot, as an example, of the screen itself, so, using a secure testing application that can prevent some of those mechanisms I think is really important to ensure exam integrity.

Allison:   0:00
Mm-hmm

Mark:   0:00
And I would definitely consider a solution with categorization of items is possible, because one of the things that we find is that providing students with feedback on particular items is helpful, but giving them information about where they are doing well and where they are struggling, from a learning outcome perspective can actually be really impactful in how they design their own test taking strategies. So, instead of them going back and studying Question 4, instead we want them to focus on specific outcomes, whether those are ILOs or PLOs or maybe they're sitting for licensure soon and they're just really struggling with a certain learning outcome that you know is going to be assessed quite frequently. It gives you the opportunity to give guided feedback to help the student.

Allison:   0:00
For our listeners who are new to “digital assessment”, what are

Mark:   0:00
Categories can be any type of information or attribute that you want to associate with an exam item.  For example, faculty can associate any topic or topics like, let's say, metabolism, to an item. And maybe it should also be mapped to a course learning outcome, and perhaps it should also be tagged to an accreditation standard.  Any attribute or metric faculty want to be able to analyze and quantify about an assessment can be turned into a category, so I advise listeners to look for a digital assessment solutions that allow for faculty to gain insights beyond the assessment score and really dive deep into the details of what's being learned and how well it's being learned.

Allison:   0:00
Great point.

Mark:   0:00
And if I could add to my answer, there's a lot of really creative ways in which you can use categories beyond just looking at institutional learning outcomes or accreditation standards.  Some programs and faculty members like to tag their items with their own name, so they can track and measure performance for items that they have personally authored.  I've also seen a lot of institutions tag items for experimental purposes, if they're trying something new, or they’re trying to tweak an approach to see how that particular exam item performed in relation to other exam items as well. So, categories can be as extensively used as your imagination will allow.

Allison:   0:00
Mark, what kind of items do the most robust digital assessment platforms have?

Mark:   0:00
Yeah, so you'll see some variances, actually between different digital assessment platforms. I think that there are the true and trusted multiple choice, true/false, fill in the blank, essay style questions. You'll also see that there has been an expansion in recent years towards more alternate question types like “hot spot” items, which gives the ability for students to drop a pin on an image in order to show understanding. You'll also find that there's a lot more multimedia that's included now. So, video, audio, as an example. What I like about the ExamSoft platform is that because the exams are offline, you don't have to worry about things like buffering, if you have a large video included on the assessment. So, there are going to be some variances but there's more similarities than there are differences in regard to the question type, it's more about how they're applied and delivered, and then some of those other features that can be built around those items, like questions grouping, and disabling backwards navigation so, you can dictate the direction and the style in which the student completes the exam. The other thing that I think is important is being able to help filter and understand feedback from students as well as just getting raw performance, and I think that being able to collect data is important easily and it's also really important to be able to interpret the data easily, you know. One of the things that people sometimes look at in assessment solutions is the type of data and reporting that's available and everyone has a different speed when it comes to data and reporting.  So, some folks are statisticians, or you know, they play one on TV, so to speak, where they just love diving into the data and I'm probably one of those people, not the statistician, but the one that you know likes to pretend to be.  But I love data.  I could spend all day looking at spreadsheets and looking at reports and slicing and dicing.

Allison:   0:00
You and me both.

Mark:   0:00
And I know that's not necessarily the case for everybody, you know. So, we provide you that full gambit. At ExamSoft and I would assume of other assessment platforms that exist out there, that if there is going to be that rich data set, it has to accomplish two things. So, one has to be detailed enough to allow for the deep dive and the slice and dice of the data. It also has to be operational enough for somebody that has zero experience with statistics, to come in and take a look as something and quickly identify where they need to focus their time. Whether it’s on remediation, or if there are particular items that maybe not be up to snuff, or were mis-keyed, or whatever the case might be. So, you know, you want to find something that allows for you to accomplish something with the statistics. But you also want to make sure that it's robust enough, where you feel like that there is value in using the data. I've worked with a lot of folks over the years, who have also come from platforms where they had such an immense amount of information that they didn't know where to begin.  So, you want to strike that balance it has to be somewhere between no information in all of the information in the world.

Allison:   0:00
Hmm

Mark:   0:00
Really, it's about “how can I maximize my time and understanding of the statistics in order to make informed data driven decisions?”

Allison:   0:00
Mark talk to us about data and insights for students, faculty, and program administrators.

Mark:   0:00
Yeah, so again, when we're looking at data, we want to be able to operationalize it. We want to actually be able to turn it around and turn it into something. A good example of this, is when you're looking at individual student performance, it allows for you to start evaluating trends in performance, and to help remediate early and often. I think that that's a real key way that you can utilize the data.

Allison:   0:00
Yeah. With a digital assessment solution, you can get information about a ton of stuff. Starting with the assessment itself. You can get a measure of how reliable your assessment was. An item report, for every single item in your assessment, indicating how difficult it was, whether or not it was discriminating, whether all of your distractors for multiple choice questions were distracting or plausible to students. You can get information about the student performance. So, what categories and subjects did all students struggle with, and what subjects did each student struggle with individually? You can get information about their performance, their rank in the class, as well as being able to share with them which items they missed, and why.

Mark:   0:00
Absolutely. I think when you're looking from a longitudinal perspective, being able to understand how often you're assessing specific learning outcomes, where you're assessing those outcomes, and then with what proficiency level are the students demonstrating competency, can all be impactful, in terms of doing your curricular tuning or curricular review. So, if you're noticing that you really should be on pace to be assessing a certain outcome “X” number of times and you're falling way short, well hey, now you have the opportunity to make changes.

Allison:   0:00
Mm-hmm

Mark:   0:00
Or if you're over assessing something, same thing. You know,  it's about an opportunity to really modify the way that you're doing assessment in order to better assess the things that are going to be most relevant for that particular course or unit or even if we're looking at it longitudinally, from ensuring that we're hitting our benchmarks for accreditation purposes as well. Keep in mind, too, that we are talking about data across a multitude of different assessment types. Not only high stakes assessments, but you can utilize categories for your audience response exams, for your open book exams, as well. So, being able to start closing the gap and ensuring that you're assessing at all points throughout the student experience is also really important. It can't just be those high stakes assessments you're getting data points twice a year. It really needs to be consistently done throughout the program, as well.    

Allison:   0:00
Mark, help us wrap up the must haves for a digital assessment platforms. In the time remaining, what is the one final, critical, piece to consider?

Mark:   0:00
Last but certainly not least, one of the other things I would consider to really strongly look is proctoring. So, you want to look for a solution that's going to be, again, a one-stop-shop, but again offer you all of the different features and benefits that you might look for in an assessment. Even if you're not using them every day. Even if you're not using them tomorrow. It's good to know that you're building something that's going to be future proof and focused on whatever need may arise in the future, although you might not necessarily anticipate needing it today.

Allison:   0:00
Mark, in a few weeks, you’ll join me again to talk about proctoring, but coming up in Pedagogo, I’m joined by Directors and Provosts talking about what they are thinking about and planning for for fall to be successful. For many of the leaders, reliable proctoring services are part of their plan for success. Both finding the right proctoring service and prioritizing space in the budget for this new expenditure.  Briefly, what are the proctoring options available to faculty? Ranging from none at all, to having a live pair of proctoring eyes on each student.

Mark:   0:00
With proctoring and the different options, and it tends to be an educational opportunity to ensure that everyone understands what exist out in the marketplace. So, you do hear a lot about live proctoring. As an example, or you have literal “eyes on glass”.  The challenge with having live proctoring is that it also requires for there to be live proctors available at that given moment. So, that can certainly create a challenge, especially given the increase in demand that we've had over the last few weeks.

Allison:   0:00
Right    

Mark:   0:00
The other option that we find our other avenue that we have seen in the marketplace is the AI route. Where you're going to have that video reviewed by the AI, similar to what we do here at ExamSoft. The challenge of relying on a solely AI remote proctoring solution is that there's going to be a lot of false positives. So, I was talking to somebody that had a student test within a clinic. They had an EKG in the background, so every single time it went off, it was creating a false flag in the incident, or in the AI report, that they were receiving from that particular vendor. With ExamSoft we are able to provide the best of both worlds, by creating a recording of the student and then utilizing the AI to flag any potential behavioral anomalies. But we don't stop there, so then we take an actual person and have them review the entirety of the exam experience, to write up a customly written report. So, it gives a peace of mind and confidence, and also allows for proctoring to be done on demand, without the necessity to preschedule with a third-party vendor.

Allison:   0:00
So, Mark for all our listeners, who are listening amid COVID-19. If they adopt a digital assessment platform in the midst of COVID, will their time and money be well spent post-pandemic?

Mark:   0:00
So, we're having a lot of folks reach out to us with immediate and urgent needs to conduct testing, for, let's say, the finals that are coming up. I think that a lot of folks are taking a two-prong approach here. I think that's the right approach. The first one is how do we address our immediate need? The second part of that is, how do we build something sustainable for the future, where we're actually building up on things that we can do now and continue to find value after this pandemic passes? I think that with ExamSoft at least, I can say with 100% certainty, that you're going to be investing into something that's going to help you long-term. You know the best day to start collecting data is yesterday, and the second-best day to start collecting data is today. Once you start building that data set, and you start to facilitate that reporting process, once you start getting students utilizing computer-based technology, where you can help set guidelines and best practices, and help enforce different processes that are going to provide you with greater insight in terms of the student experience, while also ensuring exam integrity, you're going to find that you're going to want to make this change globally. Not just for the right now, but for the long haul as well. So, I'm looking forward to seeing how people are able to best accomplish their goals. Not only from the short-term, but how we can really change the mindset of how assessment works, more globally in education as well.

Allison:   0:00
To wrap up digital assessment platforms with a pretty little bow, let's recap some of the nice to have a needs to have for a digital assessment platform. We talked about the nice to haves of features that ease faculty and student burden; a robust platform with novel question types that programs can grow into, along with robust audio and video incorporation; and administrative conveniences like integration with the LMS and single sign-on. We also talked about the need to haves, starting with security;  the importance of selecting a platform with categories, so students can understand where they can take ground and where they're succeeding on a categorical basis, not just on a question by question basis; platforms that give you robust data and insight that  will help you remediate students, plan for accreditation, and inform your teaching; and finally the importance of platforms with seamless integration with proctoring services that are available on your schedule, that don't give too many false positives, and that provide faculty with evidence, instead of making the decisions for them.  That's it for us. We hope you learned a lot.  Stay tuned in the coming weeks as I talk to guest hosts including the Assistant Dean at the University of British Columbia, both the Director of Assessment,  and the Provost and Vice President of Academic Affairs at Arizona College, and the Provost and Professor of Psychology at Barry University on how they're thinking about fall,  what to plan for, and how to be successful.  I'm your host Allison Case, and I’ll talk to you then.

Britt:   0:00
“Pedagogo”, brought to you by ExamSoft, the assessment software that keeps security and integrity in your exams, while providing you actionable data for your outcomes. When creating the tests seems tough, ExamSoft gives you rainbows so you can pass your students with flying colors.

Keeley:   0:00
This podcast was produced by Allison Case, Mark Spitzer and the ExamSoft team.  Audio engineering and editing by Adam Karsten and the A2K productions crew including me, Keeley Karsten.  This podcast is intended as a public service for entertainment and educational purposes only, and is not legal interpretation nor statement of ExamSoft policy, products, or services.  The views and opinions expressed by the hosts or guests of this show are their own and do not necessarily reflect the views of ExamSoft or any of its officials, nor does any appearance on this program imply an endorsement of them or any entity they represent.  Additionally, reference to any specific product, service, or entity does not constitute an endorsement or recommendation by ExamSoft.  This podcast is the property of ExamSoft Worldwide Inc., and is protected under US and International Copyright and Trademark laws.  No other use including, without limitation, reproduction, retransmission, or editing of this podcast may be made without the prior written permission of ExamSoft.  

:   0:00


:   27:24