On 25 March 2024, the Department for Science, Innovation and Technology (DSIT) published new guidance on procuring and deploying AI technology and systems responsibly in the HR and recruitment sector. The guidance is aimed at a non-technical audience.

Developed with feedback and contributions from the ICO, EHRC, Ada Lovelace Institute, APSCo UK, Autistics, CIPD and REC, this guidance warns that though AI-enabled tools are great assets to the recruitment process, these technologies also expose companies to novel ethical risks. To help combat them, it highlights various assurance mechanisms that can be used at each stage of procurement and deployment. These mechanisms can provide organisations with the tools, processes and metrics to evaluate the performance of AI systems, manage risks and ensure compliance with statutory and regulatory requirements.

The guidance further highlights that the procurement, deployment and use of AI should adhere to the UK government's five regulatory principles:

• safety, security and robustness;
• appropriate transparency and explainability;
• fairness;
• accountability and governance; and
• contestability and redress.

Risks of AI in recruitment

The guidance emphasises the risks that come with the use of AI in recruitment – the biggest of these being unfair bias and discrimination against applicants, particularly those whose age, disability, socioeconomic status or religion mean they may not be proficient in, or have access to, technology. While AI has great potential, its use should be balanced against the risk of digital exclusion of these groups of individuals. Work should be done to combat this risk.

An article in the Financial Times entitled "For young people, the job search has never been so miserable" by Margaret Heffernan gives an insight into some of the problems of relying on AI to sift candidates in the recruitment process. It refers to a 2021 study by Harvard Business School that revealed 90% of employers use automated tracking software to sift through applications despite most acknowledging that these systems vet out qualified candidates simply because they do not match the precise criteria in the job description. With automated systems also not often giving feedback, many potentially good candidates are being missed at early stages of the recruitment process.

To help organisations minimise these risks, the guide contains an annex with a non-exhaustive list of AI tools used in recruitment processes and their associated problems.

Assurance mechanisms

The guide identifies various assurance mechanisms that should be used at each stage of the procurement and deployment process, and provides examples and links to third party resources of templates for each of them.

Such mechanisms include:

• AI governance framework: Companies should create a framework setting out how AI will be embedded in an organisation's existing business functions, who will be responsible for the AI system, the organisation's transparency commitments and a risk management framework showing how user feedback will be addressed.
• Impact assessment: A process to anticipate the wider effects of an AI system on environmental, equality, human rights, data protection and other outcomes. There are two main types of impact assessment: algorithmic (which considers the short- and long-term impacts of an AI system on areas such as data protection, accessibility and bias) and equality (which focuses on equality outcomes).
• Data protection impact assessment (DPIA): A process for identifying and minimising data protection risks where personal data is being input into AI systems.
• Bias audit: A process for assessing whether there is bias in the input data or outcome of a decision or classification made by an algorithmic system.
• Performance testing: An assessment of the performance of AI systems against predetermined quantitative requirements.
• Risk assessment: A process running either as part of, or parallel to, an impact assessment which identifies and determines how to mitigate potential risks.
• Model cards: A standardised reporting tool to capture key facts about AI models such as the training data they have been fed.
• Transparency: The use of AI systems in recruitment should be clearly signposted to applicants and potential applicants along with, where possible, how specific limitations of the systems might apply to them. This allows for contestability and reasonable adjustments to be made where necessary.
• Training and upskilling resources for employees.
• User feedback to ensure issues can be reported.

The mechanism(s) needed will vary depending on which stage of procurement or deployment of the AI system an organisation is at. The guide breaks down which of these may be helpful at each stage with some useful examples.

Stages of procurement and deployment

Alongside this, the guide also explains what organisations should consider at each stage of procurement and deployment of AI systems.

Considerations prior to procurement

Prior to procuring any AI system, organisations should consider their desired purposes and output and whether the system's functionality can produce them.

Consideration must also be given prior to procurement to applicant accessibility requirements and any reasonable adjustments potentially required to ensure applicants with protected characteristics are not disadvantaged. Failure to do this may lead to organisations being found to have breached the Equality Act 2010. The guidance additionally encourages organisations to consider whether their AI system will involve solely automated decision-making and, if so, whether a data protection impact assessment should be completed prior to procurement.

Considerations when tendering for an AI system

Moving on to the stage of seeking tenders to supply an AI system, organisations will need to consider the accuracy of the system, any risks that accompany it, and its performance and capabilities. Transparency again is key here to ensure organisations properly alert candidates to any risks with the system that arise from these considerations.

Considerations before deployment

Before deploying a third-party AI system, the guide recommends that organisations run a pilot with potential users. One objective of such a pilot should be to ensure that the employees who will be using the system come away from the pilot clear on the purpose, functionality and outputs of the system, and that feedback is collected from them on any initial issues.

Further, an assessment should be carried out against equality outcomes to identify any learnt bias or inaccuracies produced by the system towards groups with protected characteristics under the Equality Act 2010. Not only will this allow any such bias to be addressed early on, but it will also help to identify any reasonable adjustments that may need to be made for applicants with protected characteristics so that these may be implemented before the system is rolled out in full.

Once the AI system is fully rolled out, ongoing monitoring will be necessary to ensure any errors or bugs that reduce the system's effectiveness can be fixed. User feedback should be continuously sought to help with this.