A Tactical Guide For Transitioning from Paper-based to Online Testing

So, you’ve recently purchased an Online Testing Platform that you hope will enable you to transition your organization from paper-based testing to online testing. What next? There are lots of questions to be addressed and while no two testing organizations are exactly the same, there are many issues that are common. With this blog, we will try to provide a tactical guide for planning and executing your transition from traditional paper-based testing to delivering tests online.

The challenges of moving away from Paper-based assessments are numerous. Some are related to logistics and some are related to psychometrics. It is important to identify these challenges and plan the steps necessary to overcome them in order to achieve success. Here we look at a stepped approach where you can continue to use your organization’s proven item bank of questions while making this transition. This approach will ensure test integrity while switching to online and also provides time for working out the logistics first. We also recommend short pilots at various stages in order to make the overall transition process smooth.

Below are some of the key topics that you should consider with your team as you build and configure your Online Testing System –

• Item Authoring and Item Banking
  1. Question types and considerations for changes.
  2. Randomization options and impacts
  3. Workflow/s for building and approving questions/tests
  4. Use New/Existing Metadata for Items
  5. IRT impact during the transition
  6. Build Blueprints
  7. Create templates for Test Construction
  8. Create templates for Test Delivery
• General
  1. Manage Roles & Privileges
  2. Ensure the security of Items & Data
  3. Ensure the application branding is in line with your organization’s branding guidelines
  4. Confirm how candidates will access the tests (through a portal, link, etc.)
• Test delivery
  1. Test from any location or specific computerized locations
  2. Identification of testing sites
  3. Secure delivery options
  4. Logistics for connectivity
  5. Distribution of information related to tests (i.e. list of test takers for proctors, any supplemental forms, etc.)
  6. Randomization of questions/responses
  7. Proctor training–In-person proctoring or remote proctoring
• Report/Analytics
  1. Question types and impact
  2. Gathering new data
  3. Checking for cheating

As you work through your logistics consider an approach to rollout for your live online testing program.

Develop a video tutorial for candidates to understand navigation and screen functions.

Internal Pilot test using all materials (except actual test) that would be used by your real test takers.

• Create a “real” test for staff in your organization not directly involved in creating tests. Or you can use a focus group if you have access to one.
• If possible, use one of the sites you plan on using the actual test; or as close as possible in setup.
• Set up a few common situations (workstation with a flaky internet connection, bad mouse; lefty in the room) – always a good idea to have a couple of workstations set up for lefties.
• Have the proctor give the same instructions as would be used on test day.
• Have test takers complete the test as would be done on test day.
• Give each person a note sheet for them to write down questions. Only answer questions in person that you would in the test environment.
• Proctor the test using the software that will be used during the actual test. Have the proctor try some actual managing of the room situations such as pausing the test for a person to go to the restroom; pausing the test for an entire room; adding time for a person/group/all.
• For the test content consider randomizing questions or answers to see how the scoring works.
• Score the exam and review the results.
• Survey the test takers and hold a feedback session.
• Make adjustments

Schedule and hold pilots with real candidates that

• Require candidates to log in ahead of the test to alleviate any login issues at the site.
• Require candidates to review the video tutorial.

Characteristics of a good 1st pilot exam for scheduled, in-person, and proctored exam:

• A small set of known candidates (e.g. Promotional exam for Civil Service organizations; single class in a school)
• All test takers located near a test location
• Test site with all the same computers
• Tech-savvy proctors
• Standard multiple choice questions with proven reliability
  1. All stems and questions are somewhat short to avoid any screen arrangement issues
  2. Questions can be randomized (or not).
• Survey Candidates a week after the exam
• Get test results out as soon as possible
• Review and identify any adjustments that need to be made

Characteristics of a good 2nd pilot exam for scheduled, in-person, and proctored exam:

• A larger, but still small set of known candidates (e.g. Promotional exam for Civil Service organizations; 2-3 classes in a school)
• Test takers at multiple test locations
• Test sites with different computers
• Tech-savvy proctors
• Standard multiple choice questions with proven reliability
  1. All stems and questions and somewhat short to avoid any screen arrangement issues
  2. Questions can be randomized (or not).
• Survey Candidates a week after the exam
• Get test results out as soon as possible
• Review and identify any adjustments that need to be made.

The team’s learnings during the administration of the pilot exams will provide you with a sampling of the myriad of logistical changes required for migration to online, but also enabled you to achieve success.
From here, you can move forward with building your full program including:

• Online Scheduling
• Testing from anywhere using remote proctoring
• New question types to take advantage of your new technology and more.