SQE Independent Reviewer Annual Report 2024

Evidence was gathered through a mixture of:

• direct observation of a wide range of exam development, creation and delivery activities
• interviews with key staff at the SRA and Kaplan
• access to reports and information produced by the SRA and Kaplan
• support and advice from the Independent Psychometrician
• attendance at key meetings, including Assessment Boards and meetings that feed into Assessment Board preparations and the ongoing quality assurance of the SQE.

In order to provide a succinct overview of information gathered over the past year, this report is broken down into key activities which enable the delivery of the SQE exams:

• exam creation and production
• exam delivery and assessment
• candidate services, reasonable adjustments, mitigating circumstances and appeals
• SQE in Welsh
• standard setting, determining the pass mark and issuing results
• quality assurance.

The key processes for successful exam creation and production continue to be in place, as mentioned in the previous report by my predecessor. Assessment is viewed as an art, as there is no such thing as a perfect assessment as the various components of assessment utility, as defined by van der Vleuten – reliability, validity, acceptability, feasibility and educational impact – need to be balanced.

A crucial aspect of validity is that the assessments require the appropriate knowledge and skills that a day one solicitor is required to have. An appropriate mix of reliability and validity is observed across the single best answer multiple choice questions in SQE1 and the mix of written and live observed stations in SQE2. The diversity of item types is a strength of the assessment but does make them complex and technically demanding to create and deliver, especially given their high-stakes nature.

Over the past year, the increased demand has resulted in more assessment days within each sitting (and therefore multiple forms of each SQE1 assessment) and more sittings overall. Kaplan has expanded its academic team to deal with this increased demand. While observing exam activities, I have met multiple members of this team and found staff to be knowledgeable and committed to ensuring assessments are as valid and reliable as possible.

The SRA engages subject matter experts (SMEs), who are individuals who are qualified solicitors and tasked with providing assurance to the SRA that the assessment meets expected quality standards. They are trained in their roles to provide constructive feedback to the SRA, which is then shared with the academic teams at Kaplan.

There was previously a recommendation for SMEs and Kaplan's academic staff to have a better common understanding of the exam creation process and the rationale for the final decision-making process. This year efforts have been made to facilitate this understanding, and it is recommended to keep SMEs up-to-date with the exam build process and any future enhancements to this.

Observations over the past year highlight the importance of clearly defining the SME role and where final decisions lie. Where there is some difference in opinion between the SMEs and the Kaplan academic team, ensuring all parties understand where the responsibility for finalisation of exam items lies would help define roles and responsibilities.

There has been a change in the point at which SMEs provide input for SQE1. Where they previously provided feedback on live exam items, this has now shifted to feedback on items that enter the question bank. This has resulted in more questions being reviewed, so the expectation on responses to this feedback needs to shift accordingly.

The feedback process between the Kaplan academic team and SMEs continues to evolve, and recent improvements include encouraging a mixture of written feedback and comments on items, in addition to the opportunity to discuss feedback. I recommend that both groups continue communicating effectively around the provision and discussion of feedback, and that SMEs are kept updated on exam build processes.

There has been progress in streamlining the process in which SMEs offer feedback and responses are then provided by the Kaplan academic team. Besides commenting on exam items, SMEs have a wider understanding of the assessment process through their discussions with the academic team and observations of live assessments, calibration and marker meetings. I recommend that SMEs have oversight of how some exam items perform by the sharing of post-test metrics, as this provides a complete overview of the exam cycle. This would allow them to see how items that they have commented on have performed, and it may help contextualise their input and may help refine the feedback process in the future.

The SQE comprises two parts:

• SQE1 requires candidates to sit two assessments, each are 180 one-mark questions with no negative marking, assessing functioning legal knowledge (FLK), which require the candidate to select the single best answer out of five possible answers.
• SQE2 requires four oral assessments, taken at a small number of locations across England and Wales, plus 12 written assessments taken at Pearson VUE test centres worldwide.

During the period covered by this report, there were two sittings of the SQE1 and four sittings of the SQE2 exams, of which I was in post for the delivery of the most recent three SQE2 exams.

SQE1 and SQE2 written exams took place in many assessment centres in England and Wales, as well as international venues across Africa, America, Asia, the Middle East, Europe and Australia. For the vast majority of candidates, the exam delivery went without issue. A very small number of candidates experienced technical issues when using a Pearson VUE test centre.

SQE1 exams enable computer marked candidate responses, resulting in a robust, effective and reliable method for assessing the FLK. SQE2 exams require examiners, who are a mix of qualified solicitors and trained actors, to be versed on a common standard of marking and use their professional judgement when applying the marking criteria.

With the exception of SQE2 oral exams and a small number of candidates who have arrangements in place for a particular requirement, usually to support a reasonable adjustment, all exams are delivered via computer-based testing at Pearson VUE test centres. This was effective in 2023/24. However, given the large number of candidates across a wide range of sites, a small number of on-the-day delivery issues were inevitable. These issues mainly relate to technical computer issues or unforeseen venue issues, such as fire alarms.

With larger assessment windows, candidates suffering an assessment delivery failure are more often than not able to reschedule their exam within the same window, so disruption in these instances is minimal. The Mitigating Circumstances policy was updated this year to include consideration of assessment delivery failures. This has proved successful at providing candidates with a clear set of options and is allowing them to make more informed decisions regarding any rescheduling. In the vast majority of these cases, the impact of this was generally minimal and quickly managed so that candidates could proceed with their scheduled examination, sometimes with a bit of a delay.

In the 2022/23 report from the previous Independent Reviewer, the absence of a spell checker function was highlighted, and it was an area that I explored over the past year. Without this function, the validity of the SQE2 exams is hindered as they do not accurately replicate the context in which a day one solicitor would operate. To account for this in the marking process, additional guidance is given to markers to ensure candidates are not unfairly penalised for spelling errors that would have been picked up by a spell check function and to ensure the approach when faced with spelling or grammatical errors is consistent across markers. This seems to be an adequate temporary stopgap, but the only way to eliminate the risk of crediting candidates who cannot communicate at the appropriate competency level is through the provision of a spell-checking function.

At the time of writing, this remains a priority of Kaplan, but there is no clear timeline for the rollout of this function. Pearson VUE's computer-based exam delivery system is used by a range of clients with varying requirements, and their system developments are based on overall needs and can take some time to implement. I understand that there are currently two versions of the delivery system; there is no difference to the candidate-facing interface. But it is the newer of the two that may offer a spell check function soon as Pearson VUE have completed a project to support spell check on this version.

The main hurdle preventing the SQE2 exams moving across to this newer version that supports spell check is a delivery setup with a specific connectivity requirement. Not all test centres currently have this setup. There is no clear timeline at this stage of when all centres will have this available. I recommended to continue to highlight this as an important requirement to implement as soon as reasonably possible and that Pearson VUE consider this as a top priority for consideration.

SQE2 oral exams were delivered at test centres located in Cardiff, Manchester, Birmingham and three venues in London. The delivery of the assessments at these sites is fully managed by Kaplan. SQE2 oral assessments are logistically complex, requiring the assessor and candidate to be face-to-face in a space under appropriate exam conditions, with quarantining in place for the multiple sessions across the day to ensure no contamination of exam content.

Over the past year, I visited the oral exam venues in Birmingham, Cardiff and London (Kaplan Islington) and spoke to Kaplan and SRA staff to receive their feedback about other locations. All venues are of an extremely high calibre. Those that are purpose-built for delivering in-person skills-based performance assessments meet the needs of the SQE2 oral exams. Those venues that are intended for more generic purposes undergo an extensive set-up process to ensure they are appropriate and provide a secure, efficient and professional environment for the delivery of these exams.

Oral exam venues are well-staffed to ensure that the assessment runs smoothly, that adequate calibration occurs at each venue and, most importantly, that adequate calibration occurs between the different venues running on the day. This gives assurance that the candidate experience and the standard applied to the marking process is uniform across venues.

The October 2024 sitting was the first time that marking was solely carried out on tablets for the oral assessments. It is good to see the continued introduction of technology where possible in the marking process, as this reduces the possibility of invalid or incorrect marks and can provide a live view on exam progress.

I observed several calibration sessions for both solicitor and actor assessors for the SQE2 written assessments. These calibration sessions are run online which is an appropriate and efficient way to carry out this exercise, with a mix of large group sessions and breakout rooms to cover the calibration objectives. As with any online meeting, there are inevitably some connectivity issues that affect a small number of participants. Attendance is monitored to ensure those who attend are fully participating and present for the majority of the session.

The way in which calibration is run may depend on the session facilitator and the nature of assessors' backgrounds. Some differences were noted in the degree to which facilitators would try to actively engage participants, with various requests to turn cameras off/on during the session and asking specific individuals for comments compared to open calls for comments. I recommend that facilitators leading calibration sessions continue to actively engage participants to ensure the calibration objectives are met by all.

Candidate services processes work well, and Kaplan has an excellent approach to obtaining candidate feedback. The 2022/23 report highlighted concerns raised by a minority of candidates regarding the booking process and mentioned that it would be featured in the next annual report.

As candidate numbers have been increasing since the introduction of the SQE, the system has previously struggled to deal with the volume. Over the past year, the booking process was enhanced to allow candidates to register via a link they are emailed, rather than waiting in an online queue like in previous years. This seems to take away the stress and time it takes to book a seat. Quantitative candidate feedback showed a marked increase in satisfaction with the booking process, and qualitative feedback from candidates who have had experience with both processes explicitly praised the changes. If challenges are encountered, the Kaplan team provides appropriate solutions as quickly and efficiently as possible.

The team is constantly monitoring the demand for spaces and the geographical location of venues to ensure candidates get as close to their preferred location as possible. There was consideration of new venues for the oral assessments in the north of England beyond the Manchester venue. However, candidate demand for seats has been focused on London, which resulted in the addition of a third London venue for the most recent SQE2 sitting.

There is feedback from candidates regarding preparation materials and the desire for more practice questions. This is common with candidates preparing for any high-stakes assessment. Since November 2023, the number of SQE1 sample questions has increased from 90 to 170, with the most recent release of 40 being in November 2024. Additional practice content for SQE2 including 3 sample oral videos has also been made available during the past year and current samples have been refreshed. As question writing is an ongoing process, there should be continued emphasis throughout to develop the bank of sample questions. There is also the possibility of providing guidance to empower the training providers on the SQE1 item style, so they are able to provide their students with further preparation material.

My observations on the SQE2 oral assessments were noted above, but it is also worth noting the candidate experience under this section. In the venues observed, the staff were extremely professional and polite in candidate interactions. Candidates were kept informed of the assessment processes and timings at all points throughout the day, and the final group of the day who had the longest sequestering time received a briefing from the Chief Marshal. In a candidate focus group that I attended, their positive interactions with staff during their oral assessments was noted, so this is to be commended.

• Reasonable adjustments (RAs) are offered to candidates that request and need them. The most common adjustments are:
• extra time
• sole use assessment room
• access to medicine/snacks/water during the assessment.

Kaplan treats each candidate's request individually, and further provisions - in addition to the above - are also offered, provided adequate supporting evidence is available. The adjustment plans are generally well-communicated to candidates in advance ahead of the assessment. A very small number of candidates reported some confusion on the day of the assessment, particularly for the SQE2 oral exams, where they were unclear whether the instructions and timings provided had already taken into account their RAs.

The introduction of the longer assessment windows for SQE1 allows candidates to choose how many days apart they sit their FLK1 and FLK2 exams. If a larger gap is favourable for some RA candidates, they are able to request this. This was explicitly requested in some candidate feedback from previous years, so it is positive to see this option available. Beyond this, the RA process appears to have worked well. Outcomes for RA candidates compared to the overall cohort continue to form part of the data reviewed following each examination sitting, and no trends were observed in the performance of RA candidates compared with the rest of the cohort.

Mitigating circumstances claims can be submitted by candidates who believe they have encountered a material disadvantage while taking an exam. The majority of candidates who made a claim submitted it under 2.1(a), citing a "mistake or irregularity in the administration or conduct of the assessment".

I observed a portion of a mitigating circumstances panel, which involves the thorough consideration of each claim. These panels are scheduled across multiple consecutive days to ensure adequate time to discuss each case, align it with guiding principles and reference past decisions where available to ensure that consistency is maintained over time. The outcome from each discussion is an accept or reject that is ratified by the Assessment Board. Over the past year, there is a declining trend in the volume of mitigating circumstances submitted.

Candidates are able to appeal the outcome of their assessment on the grounds of either:

• There are mitigating circumstances which could not have been put before the Assessment Board before it made its decision.
• The decision reached by the Assessment Board or the way that decision was reached involved material irregularity and/or was manifestly unreasonable and/or irrational.

Over the past year, across SQE1 (January 2024 and July 2024) and SQE2 sittings (October 2023, January 2024, April 2024 and July 2024, there were 217 First Stage Appeals submitted (for SQE2 April and SQE1 July). 44 were upheld.

The most common reason for an upheld appeal was related to late Mitigating Circumstances submissions that could not have been put before the Assessment Board. From the information reported, the process and policy were appropriately followed, and cases were given full consideration.

The 2022/23 Independent Reviewer's Annual Report thoroughly reported on the pilot undertaken and further processes to prepare the SQE1 and SQE2 for delivery in Welsh. I did not directly observe any of these, as I understand the assessment is ready to deliver should a candidate request it. To date, no candidates have taken the SQE in Welsh, but Kaplan is aware of interest for 2025.

In my observation of the SQE2 oral exams in Cardiff, all candidates took the sitting in English, but there was evidence that the Welsh provision is ready. The layout in this venue allows for a small number of Welsh candidates to be embedded within the existing English operation. And there is capacity for a dedicated Welsh circuit if the demand exists.

From interviews with key staff members, there are currently no training providers that offer full SQE preparation in Welsh. This was fed back directly from prospective SQE candidates to be a key contributing factor as to why there have been no requests to sit the SQE in Welsh. It is hoped that there may be some demand for Welsh delivery in the near future.

Decisions as to where to set the pass marks are clearly outlined in processes and policies. The basis for these processes and policies is from well-established standard setting techniques widely used in other high-stakes professional qualifications. The processes are supported by the robust analyses carried out on the psychometric data and comprehensive reports to support the Assessment Board in determining the pass marks. The outcomes appear to be fair and defensible.

I observe the Assessment Board and have sight of the reports produced by Kaplan to support the preparation. These reports contain extensive psychometric data that provide item and station level analysis in addition to measures of overall test performance.

The Kaplan academic team review the item level data to highlight any potential items that may need further scrutiny due to performance. Options for intervention are considered in meetings ahead of the Assessment Board, where the technical data is scrutinised, and further information or analyses can be requested ahead of the formal Board meeting to help the Board to make the most robust decisions possible. I am satisfied with the processes and conduct of the Assessment Board.

The January 2024 SQE1 sitting went through the above processes and had results issued on 14 March 2024. Approximately two weeks after this, an error was found that highlighted a number of false negatives – candidates who were told they had failed when they had passed. The psychometric processes were robust in terms of analysis carried out to determine individual scores; the error was in the application of the rounding policy when making the final pass/fail decision.

The January 2024 SQE1 was the first time a forms-based assessment was delivered. Under this process, one of multiple test forms is randomly allocated to candidates. This allows SQE1 to cater to a larger number within each sitting, meeting the growing demand. This adds a degree of complexity to the standard setting process as each test form needs a pass mark that reflects the standard of minimal competence, ensuring that the level of knowledge required to be deemed competent is uniform regardless of which form is sat. In order to have comparable scores across the various forms, scaled scoring was introduced. The error was a result of the rounding not being performed in line with the published policy.

This matter was rectified in as efficient a manner as possible. As soon as the error was found, I was briefed on the issue with key SRA and Kaplan staff. An urgent Assessment Board was convened, with the action of recalculating and reissuing all results being the agreed outcome. The corrected candidate results were then issued on 15 April 2024. Before releasing these corrected results, a three-step checking process was carried out to ensure correctness.

Once the corrected results were issued, a thorough review process was undertaken to understand reasons why this occurred and to prevent further errors in the future. An independent external reviewer was commissioned to look at processes and how they can be improved to reduce the risk of future errors. The key finding was that the main focus of discussions and meetings prior to the release of results was on the calculations and psychometric processes, and there was a failure to look at whether a published policy was being applied correctly. Kaplan has taken steps to ensure policy decisions are adequately considered as part of the process between exam delivery and results issue. They have also introduced new change management processes and have enhanced organisational capability.

During 2023/24, Kaplan has demonstrated comprehensive quality assurance procedures, and even within my short tenure, I have witnessed continual improvements in processes within a single year.

Effective quality assurance checks continue to be in place and feedback from all key stakeholders drives improvements. The January 2024 SQE1 results error proved to be a learning point, and the follow-up actions as a result of the review will only strengthen assessment quality.

A significant improvement I have witnessed over the past year is dedication to the ongoing quality assurance of the assessment over time. A new process this year consists of meetings dedicated to exploring issues that are not related to any single SQE sitting and its associated results, which is the purpose of the previously mentioned Assessment Board. This process allows for time outside of the pressured results-determination process to consider overarching issues and give some dedicated time to the development of the assessment. This will ensure that future developments are proactive and considered in light of assessment best practice and educational theory rather than reactive issues arising from individual sittings. The first review was held at the end of 2024, and there are multiple others planned for the future in 2025. I believe the output from this process will further enhance the robustness of SQE policies and processes.

One area I explored is the issue of marker variability in SQE2. I have observed that adequate assessor training, calibration and feedback processes are in place.

SQE2 has been delivered for several years, and many assessors have participated in multiple sittings. It would now be useful to look at longitudinal feedback over time to provide a clearer picture of assessor performance beyond a single diet. I believe this would be useful for the professional development of assessors and to confirm whether significant differences are found which might highlight the need for retraining or guide selection of assessors for future sittings. I recommend that further analysis into assessor behaviour is conducted and feedback provided where appropriate.

The SQE is a unique assessment in the field of solicitor qualification and the wider legal education sphere, so there is not always precedence when difficult decisions need to be made or new processes developed. However, Kaplan is open to using external expertise, and I've witnessed this at several points throughout this year. I would encourage this external input from fields that run similar assessments, such as those in medical education or the wider education academic field. This highlights a culture that is constantly looking to improve and remain at the forefront of professional exams.

This report highlights areas of good practice and recommendations for future enhancements. A summary of these recommendations is below:

• continue to keep SMEs up to date with exam build and other assessment processes
• place urgency on the development and rollout of the spell check function as an integral requirement of SQE2
• ensure active engagement from all participants in online marking and calibration sessions
• continue to prioritise the release of sample exam items by generating items for this purpose as part of the writing process and empowering training providers to develop their own preparation material
• analyse assessor performance over time to provide feedback and inform further training and selection decisions.