This piece explains the main technologies for detecting known and new/unknown child sexual abuse material, as well as grooming, in the context of the EU draft Regulation to prevent and combat child sexual abuse.

EU policymakers are about to decide on a very significant legislative proposal: the draft Regulation to prevent and combat child sexual abuse. While the aim of protecting children from sexual violence online is uncontroversial, critics have warned that the proposal is technologically unsound.

Some of the key disagreements and challenges regarding technology arise in the context of measures to detect known (previously identified) child sexual abuse material, new (or not yet identified) child sexual abuse material, and grooming. This piece aims to provide some clarity around the capability of technology to detect abuse.

The detection of known child sexual abuse material

A crucial way to detect known child sexual abuse material is via a process called “hashing”. The detection tool creates a hash - a unique digital fingerprint - for an image. This hash is then compared against a database of hashes of known child sexual abuse material to find matches. “Cryptographic hashing” can be used to identify exact matching, while “perceptual hashing” can be used to determine whether the content is similar enough to constitute a match, for example even where the image has been resized, cropped or rotated. Hashing can also be used for other multimedia content, like videos.

Regarding these technologies, experts mainly disagree on their accuracy (how well a tool is able to detect child sexual abuse material) and security (resistance to attacks). For example, PhotoDNA, one of the main tools for perceptually hashing photos and videos, has been said to have a false positive rate of 1 in 50 billion. However, there is no independent review of this technology. Some experts have questioned this rate. For all perceptual hash tools, efficient attacks have been described that created false negatives (where the tool did not catch known child sexual abuse images because small changes had been made to them), as well as false positives (where the tool wrongly identified non-abuse images as known child sexual abuse material). Scientific literature has also shown that, if parts of the perceptual hashing tools are moved to user devices, the tools could be reverse engineered. This means that they could be decoded in order to extract sensitive information from them, in some cases even identifying specific people in the original image.

The detection of new/unknown child sexual abuse material

The detection of new or unknown child sexual abuse material poses greater technical challenges than detecting already known images and cannot be achieved through “hashing”, which requires an identified imagine. The identification of unknown child sexual abuse material can only be done through artificial intelligence, based on machine learning. Classifiers (algorithms that sort data into classes based on pattern recognition) can be trained to detect nudity, faces, colours etc. It is particularly challenging to detect the age of a person shown in the content, especially to determine whether they are a teenager or a young adult.

The tools that are used to identify unknown material are automated, but the work necessary to verify whether child sexual abuse material (CSAM) has accurately been identified requires human oversight. Assessing the impact of the introduction of these measures requires an analysis of the “precision”, “recall” and the “false positive error rate” of the relevant tools.

The detection of grooming

The detection of grooming requires the analysis of text through machine learning. Technologies for grooming detection find patterns pointing to possible concrete elements of suspicion. They indicate the estimated probability that a conversation is grooming. Flagged conversations are then subject to human review.

Regarding the accuracy of these technologies, the EU Commission states that Microsoft has reported that its tool developed under Project Artemis has an accuracy of 88%. However, Microsoft itself “recommends against reliance on this figure when discussing EU policy”, adding that it relates to “a single English-language technique trained on a small data set of known instances of [grooming]”. Moreover, there is no independent review of this accuracy level. Some technology experts have said that with text alone, “it can be difficult to get error rates significantly below 5 - 10%, depending on the nature of the material being searched for”. For 1 billion messages exchanged daily, this would mean between 50 - 100 million false positives every day. In this context, human review is completely unfeasible.

Read CRIN’s piece on the EU draft Regulation to prevent and combat child sexual abuse.

Read CRIN and defenddigitalme’s research on a children’s rights approach to encryption.

Sign up to our newsletter to stay updated and follow us on Twitter, LinkedIn and Instagram.