Marking music theory examinations is demanding but ABRSM (the Associated Board of the Royal Schools of Music) has more experience than most. By using DRS e-Marker® for electronic marking, ABRSM has improved quality assurance, automated results handling and reduced the risk of losing
Founded in 1889, ABRSM provides teachers and their candidates with instrumental
and theory examinations. Linked to the Royal Academy of Music, Royal College of
Music, Royal Northern College of Music and the Royal Conservatoire of Scotland,
ABRSM publishes music and delivers over 650,000 examinations and assessments each year in 93 countries.
Held three times a year in 200 centres around the United Kingdom as well as locations abroad, over 120,000 theory examinations are completed annually. Before introducing DRS e-Marker® technology for the UK, Malaysia and Singapore, examination scripts were manually sorted and distributed for marking.
“The whole operation was quite cumbersome and lots of risk was introduced by moving paper around,” says IT Director Eugene O’Donnell.
Although everything had worked well from a marking quality point of view, a
realisation that other examination boards were automating provided the impetus to change. “It was to do with streamlining the process and getting the results to candidates more quickly.”
An evaluation of electronic marking solutions helped appreciate the true potential. Although the marking process might be speeded up, the benefits were more likely to come from managing the quality of marking and its progress in real time. Any solution also had to cope with occasional rubbing-out residues on music manuscript-sized paper. Neither was an obstacle for DRS thanks to its specialised scanning technology.
“DRS showed us around their facilities and that engendered confidence. We were paid more attention and they cared more about the requirement,” says O’Donnell. “Upon reflection, our processes weren’t particularly robust. Partnering with DRS made our processes much more professional.”
ABRSM selected the DRS e-Marker® service for the first five grades of its eight grade music theory examinations. DRS gave expert advice about changing designs and
adding barcodes for traceability and security. An XML interface was built to link DRS e-Marker® to the ABRSM candidate database. ABRSM received good information about assessment, marking and setting tolerances from DRS consultants before running a successful pilot test.
DRS also provided the initial training for examiners who use the e-Marker® software to access scanned images. On-going support is provided by DRS while in-house seminars, computer-based training and a training system now prepare new examiners.
Completed examination scripts from the UK, Malaysia and Singapore are sent to DRS three times a year for scanning and processing. Around 70,000 papers were marked online last year by 54 trained examiners.
Thanks to DRS e-Marker’s® segmentation capability, examiners see a series of individual questions rather than whole examination scripts. The freedom from physical handling enables them to concentrate on accurate marking while their progress is regularly monitored to ensure quality standards. This has made the overall system more robust with real-time information available and consistent marking standards.
A powerful combination of seeded questions and double-marking ensures consistency. Pre-set tolerances allow the marking to stand, refer the marked question to a moderator for adjudication, or lock out particular examiners for scrutiny. Examiners benefit from the feedback while data gathering assists post-examination analysis.
“We can configure the system to take action when quality is not met. The system can lock examiners out and they have to talk to a senior examiner about exactly what went wrong,” says O’Donnell. “We can also more or less guarantee that everything is done by the deadline because we can see what’s left days in advance.”
Electronic marking has proved robust which, backed by a “high level of commitment” from DRS, enhances examination quality. The ABRSM-DRS data interface saves significant time and effort by speeding up results processing, and ABRSM is able to add more countries when required.
The UK, Malaysia and Singapore are the three largest markets. The UK accounts for 29% of the theory papers processed in this group. The reason that other countries have not been switched to DRS e-Marker® is that the numbers are currently relatively low.
“DRS is an entirely professional organisation which has excellent people who know
what they’re doing and give good advice. I think DRS has a very friendly corporate face,” says O’Donnell.