https://educationinspection.blog.gov.uk/2018/04/17/risk-assessment-process-for-good-and-outstanding-schools/

Risk assessment process for good and outstanding schools

I’m aware that some questions have recently been asked about our updated methodology for risk assessing good and outstanding schools and academies. Rather than attempt to answer every question individually via Twitter, I thought it would be better to respond to them all here.

First, I can understand why the words ‘machine learning algorithm’ gave some people a bit of a fright. But in reality, our risk assessment has changed very little since the previous methodology note was published.

Ofsted has risk assessed schools and academies for many years in order to help allocate inspection resource where it is most needed. It has never been used to pre-judge inspection grades.

The model has evolved over the years, as inspection frameworks and accountability measures have changed. The main change this time is a new statistical model, which we have found to work well within the data-analysis stage of our risk assessment process.

Like any modern organisation, we are keen to embrace the benefits of technology. But while it may sound ominous, ‘machine learning’ simply refers to a computer programme that helps us identify potential decline in a school, and that then re-jigs the underlying algorithm when inspection outcomes are known. It doesn’t mean we’re now using computers to make decisions without any human intervention, or indeed to judge schools. As before, Senior HMIs in the eight Ofsted regions will always thoroughly review the selection of schools for inspection and well-trained, experienced school inspectors will inspect on site.

So what do we use the risk assessment process for then? Well, as our handbook states, some good schools will automatically receive a full section 5 inspection instead of a section 8 short inspection. This occurs when our risk assessment process indicates that the quality of provision may have deteriorated significantly. Outstanding primary and secondary schools are of course exempt from routine inspection. However, if the risk assessment raises concerns about the performance of an exempt school, then it may also be inspected.

 The new computer model uses progress and attainment data from the Department for Education, enhanced with school workforce census data and Parent View responses, to produce scores for each school, ranging from the lowest risk up to the highest risk. These scores are on a continuous scale, so there are no thresholds that automatically determine that a school should be inspected.

Of course, inspection outcomes will always be based on the evidence gathered on site. So to avoid them having undue influence, inspectors are not given the findings of the risk assessment.

The basic ideas behind the risk assessment process are outlined in our inspection handbook. We wrote the methodology note for those of you who like to know more detail about the statistical methodology involved. If it sounds a little like the rise of the machines, I can assure you that it’s really not like that at all.

 

You can keep up-to-date with Ofsted news by signing up for email alerts. You can also follow Ofsted on Twitter.

 

15 comments

  1. Comment by @TeacherToolkit posted on

    Thanks for the update Sean. Will OfSTED confirm:

    1. Do OfSTED want to make judgements about schools working in different contexts, or not?

    2. If Machine Learning will use data from the past to predict when a school will be inspected, do OfSTED recognise that past data may be unreliable if benchmarks change - or will algorithms take this into account?

    Reply
    • Replies to @TeacherToolkit>

      Comment by Ofsted external relations posted on

      Could you explain a bit more about what you mean about "different contexts" please?
      In answer to your other question, Ofsted uses a broad range of indicators to select providers for inspection and data is one of them. Please see paragraph 9-12 of the School inspection handbook for more information.

      The new system uses progress and attainment data from the Department for Education, enhanced with school workforce census data and Parent View responses, to produce scores for each school, ranging from the lowest risk up to the highest risk. These scores are on a continuous scale, so there are no thresholds that automatically determine that a school should be inspected.

      Reply
      • Replies to Ofsted external relations>

        Comment by @TeacherToolkit posted on

        1. Different Contexts (the dictionary says) "the circumstances that form the setting for a [school] ... in terms of which it can be fully understood." E.g. working with different groups of students.

        2. Could you give an example of what a continuous scale looks like? The inspection handbook suggests "...publicly available data" which I assume is a "broad range of indicators for inspection". As you mention "data being one, parent view and also school workforce census data". What are the other broad range of indicators? e.g. school funding; retention rates; current vacancies; re-sit data etc. Secondly, "stage two involves a more in-depth desk-based review of a wider range of available information." What is this information that is placed into the machine learning algorithms?

        Many thanks for the time to clarify and support teachers on the front line of artificial intelligence data being used to predict who should and should not be inspected.

        Reply
        • Replies to @TeacherToolkit>

          Comment by Ofsted external relations posted on

          Each provider (school/academy) is unique in context. Our inspectors follow our school inspection handbook, which seeks to balance the need for consistency in inspections with the flexibility required to respond to the individual circumstances of each school. Inspectors will also exercise their professional judgement.

          A continuous measurement is one that can take an infinite number of values, and is limited only by how accurately you can measure it. In our risk assessment, the continuous scale runs from 0 to 1, a little like a probability.

          The handbook and blog post above outline the indicators used. In answer to your final question, the desk-based review is carried out by Senior HMIs who use their professional judgement and local intelligence - this isn't an automated process.

          Reply
  2. Comment by Yitzchak Freeman posted on

    Will it be possible for individual schools to find out their risk assessment score and how this was arrived at, as a tool to assist them in identifying and addressing possible areas of weakness?

    And will Ofsted be publishing aggregated data analysis on e.g. trends over time in risk assessment scores, regional variations in risk assessment average scores, variations in RA scores by sector, and - importantly - correlation between RA scores and inspection outcomes?

    Reply
    • Replies to Yitzchak Freeman>

      Comment by Ofsted external relations posted on

      We do not publish our risk assessment information on schools. If risk assessment identifies concerns or performance issues, we may carry out an inspection.

      Reply
  3. Comment by Paul Garvey posted on

    HI Sean,

    What benchmarks are used at Ofsted by HMI, using the /machine learning, to decide whether they should look at a school in more detail to determine whether it should be inspected? At what levels do you set these data triggers?

    Reply
    • Replies to Paul Garvey>

      Comment by Ofsted external relations posted on

      Ofsted uses a broad range of indicators to select providers for inspection and data is one of them. Please see paragraph 9-12 of the School inspection handbook for further information (https://www.gov.uk/government/publications/school-inspection-handbook-from-september-2015).

      Reply
      • Replies to Ofsted external relations>

        Comment by Paul Garvey posted on

        It doesn't really answer my question. I'm aware of what's in the handbook, which says little. Data is a very important element of selection for inspection, hence the development of the 'Machine Learning' algorithm. Take this example, for instance.

        If nothing else varies, from their last inspection, how far would progress and attainment data have to fall to trigger an inspection for a previously 'Outstanding' school?

        Thanks, Paul

        Reply
        • Replies to Paul Garvey>

          Comment by Ofsted external relations posted on

          The indicators that we use to select providers for inspection are set out in our handbook and in this blog. It’s impossible to quantify the exact amount that progress and attainment data would have to fall to trigger an inspection, because the final risk scores for each school are based upon a combination of factors. So two schools with identical progress 8, for example, need not have identical risk scores. This means there are no simple thresholds of particular factors that automatically determine that a school should be inspected.

          Reply
  4. Comment by Julian Sorsby posted on

    HMCI has indicated a desire to inspect more currently exempt outstanding schools and to seek a change in the law to effect such a change. Might the same be achieved by raising expected outcomes for outstanding schools in the context of current legislation?

    Reply
    • Replies to Julian Sorsby>

      Comment by External relations team posted on

      Hi, Some exempt schools are already picked up through our risk assessment process. Raising expectations would require changes to the inspection framework and handbook and that would require consultation. This would still mean that many outstanding schools, including those that have not been inspected for over 10 years, were not required to be inspected.

      A legislative change to allow Ofsted to inspect currently exempt outstanding schools would allow for the full range of indicators to be considered in the process of a Section 5 inspection.

      Reply
  5. Comment by Confused.com posted on

    Good afternoon,

    Could you confirm what would happen in a scenario where an outstanding, exempt school, was inspected following a health check and there was a possibility it would only now be good. Would that result in a 2 day inspection and full report or a letter with recommendations? Thanks in advance for any information on this.

    Reply
  6. Comment by Corporate wellness posted on

    Great Blog Thank you

    Reply

Leave a comment

We only ask for your email address so we know you're a real person