Goals
- Promote learning by delivering formative assessments
- Increase engagement by adding interaction to learning activities
- Benchmark performance before and after learning activities
Background and strategic fit
LearningBuilder is an assessment platform. One of the modes of assessment is to administer on-line, low-stakes assessments. These assessments are typically used for Practice Exams and to validate learning for situations in which organizations curate learning activities but wish to promote and/or measure engagement with learning materials. Adoption has been slow, but we have a handful of customers who use the assessment engine (ABO and ABOM). Other customers have inquired about it and it comes up frequently with prospects (e.g., NBCRNA). - May 2018.
Assumptions
- The Assessment Engine is not designed to manage highly secure tests. We want to make it annoying to hack the assessments, but not dedicate significant resources to preventing cheating.
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Create an Assessment | As an Administrator, I create an Assessment so that I can evaluate a person's knowledge. | Existing  | |
| 2 | Create a Section | As an Administrator, I create a Section of questions that groups a set of questions around common content. | Existing | |
| 3 | Create a Question | As an Administrator, I create a Question so that I can measure a specific component of knowledge. | Existing | |
| 4 | Re-Order Questions | As an Administrator, I order questions so that there is a sequential experience of questions. | Existing | |
| 5 | Create a Response | As an Administrator, I create a Response so that a person can answer a question. | Existing | Probably move to Question page |
| 6 | Specify a Cut score | As an Administrator, I create a cut score to establish the score necessary for a passing result. | Existing | |
| 7 | Specify a Competency Model | As an Administrator, I assign a Competency Model to the Assessment to govern the scope of Competencies available to assign to Questions. | Existing | |
| 8 | Define Result | As an Administrator, I manage a Results page so that the results can be tailored to the kind of assessment being taken. | Existing | |
| 9 | Publish an Assessment | As an Administrator, I publish an Assessment so that it is available to be completed. | Existing | |
| 10 | Un-publish an Assessment | As an Administrator, I un-publish an Assessment so that it is not available to be completed while changes are made. | Existing | |
| 11 | Automated Scoring | As an Administrator, by indicating the "correct" answer for each question (or by assigning each answer a value and specifying a Cut Score for the Assessment as a whole), I can create an Assessment that is immediately and automatically scored so that the user receives immediate realtime feedback about their performance. | Existing | |
| 12 | Manual Scoring | As an Administrator, I indicate that certain questions (such as essays or other complex interaction types) require human evaluation before a score can be assigned. | Not implemented yet | |
| 13 | Manage Forms | As an Administrator, I create multiple Forms within a single Assessment so that I can administer the Assessment multiple times without duplicating the Assessment Experience. | Must Have | This is a requirement for the NBCRNA |
| 14 | Specify that the Assessment has a Time Limit | As an Administrator, I specify the amount of time in minutes a person is allowed to complete the Assessment so that the Assessment is not open indefinitely. | Must Have | This is a requirement for both NBCRNA and ABO |
| 15 | Specify that the Assessment will be Timed | As an Administrator, I specify that an Assessment is Timed, so that the system records the amount of time as a person takes to complete each Question. | Must Have | This is a requirement for NBCRNA |
| 16 | Specify that the Assessment will use Confidence Ratings | As an Administrator, I specify that questions will include a Confidence Rating to understand how confident the Test Taker is in the select answer during an Assessment | High | NBCRNA has listed this requirement as Optional |
| 17 | Specify the Confidence Rating Scale | As an Administrator, I specify the Rating Scale used for the Assessment so that my ratings can use language and specificity I am comfortable with. | High | This can probably by a System Administrator setting, but I expect it to work well as a Custom List. it is probably common to the system as a whole instead of specific to an Assessment. |
| 18 | Specify that the Assessment will use Relevance Ratings | As an Administrator, I specify that questions will include a Relevance Rating to understand how relevant the Test Taker believes the content is to his or her practice. | High | NBCRNA has listed this requirement as Optional |
| 19 | Specify the Relevance Rating Scale | As an Administrator, I specify the Rating Scale used for the Assessment so that my ratings can use language and specificity I am comfortable with. | High | Same as Confidence rating |
| 20 | Answer Confidence Rating | As a Test Taker, when I answer a Question and Confidence Ratings are turned on, I will answer my Confidence Rating so that I can share my level of confidence in my answer with the Board | High | NBCRNA has listed this requirment as Optional. It is also possible that we could satisfy this requirement by adding a question to the Assessment specifically as a confidence rating for the previous question, but I fear that would get complicated, especially with respect to using Forms). |
| 21 | Answer Relevance Rating | As a Test Taker, when I answer a Question and Relevance Ratings are turned on, I will answer my Relevance Rating so that I can share my belief in the relevance of the Question to my practice. | High | Same as Confidence. |
| 22 | View Time to Complete | As a Test Taker or an Administrator, I can see how long it took to complete a sitting of an Assessment | Must Have | NBCRNA lists this as a Requirement |
| 23 | View Time on each Question | As an Administrator, I can see how long each test taker took to complete each Question. | Must Have | NBCRNA lists this as a requirement. I'm not sure where this would go. |
| 24 | Record Item Response Strings | As the System, I record the responses to each Question in the QTI standard for the Item type so that the results can be shared with any system using the QTI standard | Must Have | NBCRNA does not reference the QTI standard, but does reference "Item response strings," for which there is a standard, even if we don't follow it. |
| 25 | Record Post-Test Surveys | As a Test Taker, I can complete a Post-Test Survey so that I can provide feedback to the Board regarding the Assessment Experience. | Must Have | We may be able to do this using Workflow Questions instead of needing a feature. |
User interaction and design
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|