Researchers have produced a collision in iOS’s built-in hash function, raising new concerns about the integrity of Apple’s CSAM-scanning system. The flaw affects the hashing system, called NeuralHash, which allows Apple to check for exact matches of known child-abuse imagery without possessing any of the images or gleaning any information about non-matching pictures.
On Tuesday, a GitHub user called Asuhariet Ygvar posted code for a reconstructed Python version of NeuralHash, which he claimed to have reverse-engineered from previous versions of iOS. The GitHub post also includes instructions on how to extract the NeuralMatch files from a current macOS or iOS build.
[…]Once the code was public, more significant attacks were quickly discovered. A user called Cory Cornelius produced a collision in the algorithm: two images that generate the same hash. If the findings hold up, it will be a significant failure in the cryptography underlying Apple’s new system.
American tech media and bloggers have been shoving the valid concerns aside ever since Apple announced this new backdoor into iOS, and it’s barely been a week and we already see major tentpoles come crashing down. I try not to swear on OSNews, but there’s no other way to describe this than as a giant clusterfuck of epic proportions.
Yes, there is very little coverage, but expecting more from the media is probably not realistic.
I expect most tech journalists to be conflicted. They of course have affinity to our common interests. However if Apple says this is a good thing, and a few random people say it is not, they will probably choose the safer option.
Even though people individually would have good intentions, there would be an implicit bias to favor their favorite companies.
In game journalism, this is perceived as being “bribed”: https://www.reddit.com/r/Games/comments/sdp2p/are_game_reviewers_actually_bribed/. It is not, but there is an “access” issue. I remember seeing some outlets blacklisted for negative reviews. Same could be thought for any outlet covering a tech firm. A “Microsoft Insider” for example is unlikely to heavily criticize Microsoft and continue being an insider.
Again, this is probably at subconscious level, and not done with a bad intent.
It’s something I’ve always wondered.
Why do we place such faith in standard hash functions for uniqueness?
The first time I really thought about it was when I was first shown how GIT works. It basically says that 2 files are equal if their hashes are the same. Yes, we all operate with full knowledge that hash collisions in real life are rare. Yet, they still happen. It puzzled me why the uniqueness of a file didn’t include some actual data of the file (file size, maybe name…). Apparently, it was just always thought that those bits to store actual data of the file would be better spent making the hash-size bigger.
To me, making something like file-size a part of the hash cuts off entire vectors of attack (can’t remove or add data to a file to make the hashes the same) for example. Storing the filename help cut off any accidental file collisions that might screw up the repo at the expense that file renames might be stored as a ‘new’ object.
In this case, it seems Apple has at least designed it to expect collisions. But the details of the secondary validation system are not really known, so who knows what it really does.
Like I said, I know using hashing as uniqueness seems to be a thing we depend on in the modern age. There’s just a part of my brain that doesn’t want to accept it 100 percent. Linus would probably have a field day on me 🙂
Yamin,
Well, the idea with cryptographic hashes is that even if we dedicate all our planet’s resources to finding collisions intentionally, it would take an implausible amount of energy to find a collision. This is the principal that bitcoin is based off of, which is designed to search for easier partial collisions to control the rate at which they can be found. Of course mathematically there are infinite numbers of inputs that would hash to the same value, but since we don’t have the computational power to find them, it’s not considered a problem. Of course the odds are statistically greater than 0%, but minuscule nevertheless.
Some hashes do include the length of data (although this gets blended into the hash and isn’t really a separate field).
To the best of my knowledge nobody’s ever found an sha2 collision, but one has been produced for sha1 and md5 is broken too.
https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
Well there’s a huge difference between cryptographic hashes and the fuzzy hashes that apple and others use for images/biometrics/etc. If they were using a cryptographically secure hash, it would would completely change the hash even if a single bit were changed, which is useless for fuzzy matching applications. They are NOT using cryptographicly secure hashes, and so this result was expected and predicted. It’s a lot easier to find collisions with fuzzy hashes because they’re literally designed to have more collisions.
Cryptographic hashes are solving two separate issues: 1) they have to protect against natural collisions happening via the birthday paradox, and 2) they have to protect from code that deliberately searches for collisions.
https://en.wikipedia.org/wiki/Birthday_problem
#1 is easy, just add more bits. Consider that if you’re just interested in “uniqueness” for #1 and not security, you could use an arbitrarily large checksum, like 2048bits, which is statistically safer against random collisions than any of the SHA algorithms because every SHA-512 hash has 2^1536 times more collisions than a 2048bit checksum. But a checksum has zero cryptographic value and we can simply enumerate through the inputs that produce the same checksum.
#2 is harder and is why many algorithms are assigned a cryptographic bit strength that is less than the number of bits used.
Anyways I’ve gotten a bit off topic, haha. Just remember that fuzzy hashes are nowhere near as secure as the ones we use for cryptography.
American tech media and bloggers seem to exist only to sell stuff. It used to be interesting and although I rarely read it given the few links I’ve read during the past month or so I’ve pretty much given up on Arstechnica. Everything seems to be a well articulated sales pitch with fairly obvious blindspots.
Fun fact: All this scanning has existed in “iCloud” for some time ago, they are just moving the code from the iCloud servers to the consumer’s machine so they can save money on server infrastructure (at the expense of the user who is having their CPU cycles and battery eaten by something that doesn’t benefit the user experience).
So, all these issues with scanning were somehow ok when the scanning happened “in the cloud”. Which is the issue here: The perennial “it’s their cloud, their property, you can’t complain” chant can be used to silence a whole range of valid complaints. Even for “clouds” that come pack and parcel with a device.
It is fundamentally different when it’s happening to files at-rest on Apple’s servers, to when it happens on your own device, at least when Apple can actually acess those files.
However, from what I understand, iMessage uses end-to-end encryption making it difficult for them to perform such scanning in transit, and they’ve been playing around with end-to-end encryption for iCloud data and backups.
This measure appears to allow them to scan iMessage images locally (which as above is new) as well as yes shift stuff from the cloud to the device (possibly also allowing them to do end-to-end encryption for iCloud data too).
Once the code is on the device, it’s trivial to extend it to other applications and data beyond just cloud stored/transmitted data, as well as being trivial to extend to other types of “illegal” content (Political opponents or copyrighted materials, for example).
The “send to apple to check it” thing when there’s a potential match is also a major issue, because particularly for iMessage some stuff may be highly sensitive and subject to other restrictions (medical information, for example).
The1stImmortal,
Yes, many of us feel the risk of mission creep is high. Once apple demonstrates this capability, it will become irrelevant what apple has promised it’s users. Governments can and will come knocking to serve court orders demanding to use it, possibly even in secret.
That’s an interesting point, they could potentially violate HIPAA laws if the process got triggered on medical records. I’m not sure how many medical providers and/or patients actually use IOS devices for medical applications, but it’s possible.
@Alfman
On Android and Google in general Email and location services already fall foul of privacy law in Europe. If it’s not medical people it’s patients being swept up in corporate surveillance. Of course in the UK there’s a little known law which forces the NHS and banks and others to hand over all their information to the security services.. This isn’t shared with other agencies unless someone becomes a national security threat which meant that the flamboyant politician Cyril Smith and the celebrity Jimmy Saville got away with it for years. The security services publically admitted they knew but they were acting on the basis of strictly interpretted law which restricted use of information to security only. Although in the last instance I understand the security services warned then PM Margaret Thatcher not to give Saville a knighthood because he had “character” issues. As for lawyers I’ve noticed some lawyers breach “safe harbour” laws by slipping into their T&Cs unilaterally outsourcing some admin services abroad outside of EU safeguards. A fair few are very cavalier with email going through Google and I daresay other foreign email providers.
Given that the new scanning technology is designed to detect images of children being abused or at least children who are naked, surely a valid collision would be a photo which matched certain criteria whose hash collided with another one that didn’t. An image that isn’t a real-world image that generates a collision doesn’t reflect a fault in the system. I do worry about this because any technology they can use to spot child abuse images or footage at the government’s request can be used to spot other ‘incriminating’ material at any government’s request, but false positives are less of a worry.
Matthew Smith,
These hashes are not a 1:1 representation with the subject it is meant to represent. They’re looking at nuanced details while throwing away the majority of the information that the image is comprised of. Matching hashes means the sources share some details, but are not necessarily of the same image. A previous post illustrates this with two totally different pictures that are considered to match a specific hash.
https://rentafounder.com/the-problem-with-perceptual-hashes/
Both false positives and false negatives are a problem, so a balance needs to be carefully considered. Ordinarily you could just compare the two sources to make sure you’re not dealing with a false positive, but that doesn’t work here because apple isn’t allowed to have the originals for obvious reasons. Given the number of images we’re talking about, random collisions will be likely due to the birthday paradox, and even more likely if people set out to generate collisions on purpose.
So whatever apple’s policy ends up being, it should not dismiss false positives.
–Given that the new scanning technology is designed to detect images of children being abused or at least children who are naked, surely a valid collision would be a photo which matched certain criteria whose hash collided with another one that didn’t. —
Bad news here. We know that is not true in a useful way.
https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1
We have the existing example of a valid collision between a dog picture and a grey mess. Others have been working on methods to generate a picture that looks less and less like a dog instead look like modern art.
There is a risk that you go to a art gallery take 30 photos of modern art and get flagged.
https://petapixel.com/2017/12/20/uk-police-porn-spotting-ai-gets-confused-desert-photos/
Please note issue with this kind of hash is not new like the 2017 cases of people with desert photos end up being really heavily searched. Not because they had any porn just because they had photos of the desert.
Also the early research has found flaw to create false negatives big enough to drive a bus though. Simply crop the point of interest out the photo and you get a different hash.
Let say the information we have about the apple has is right that it scales to a 360×360 image RGB before processing.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf This is apple documention on it page 5 is critical. The match there is because its now not a 360×360 by 24 bit RGB but now a 360×360 by 256 grey-scale.
129600^256 is the total number of possible unique inputs to the Apple NeuralHash. Of course the number of unique NeuralHash is not that big there is only 2^96 7.922816251×10^26 This is simpler to create collisions than a MD5 remember MD5 2^128. Yes when you have a grey scale collisions it will be possible to color a picture over top of that does not look anything like the source image. Like a 1080p image 5.33×3 boxs for every byte the apple NeuralHash is going to use. Yes that dog grey scale collisions example could have a picture of a cat, porn, landscape basically what ever just by coloring without altering shading in a big way. Remember you have 8 bits coming out of 24 bits this gives you a lot room to play and that is without the image being bigger than 360*360.
So yes if someone is able to work out what are the triggering hash values they will be able to produce images that will trigger the system that totally look harmless and to the human eye look nothing like the targeted images.
The certain criteria for a match is insanely broad due to the image being greyscaled and scaled down to 360*360 then processed to a 96 bit number using floating point. To the point you take a photo at night of a wall and you might be generating a flag-able image just because you camera flicked the pixels the wrong way. Yes the dog collision shows you how little the collisions has to look like the original picture. Yes you can grey scale the dog and put it next to the blob mess collision that has the same value in the NeuralHash and it clear they are no where close to each other. So there are going to be lots of false positives. Worse those with images will really simply be able to make those images false negative.
Only the idiots with child-abuse images will get caught because the smarter ones who have been following this will have already worked out how to alter or encyrpt their images not to trip the detection. Yes the most likely ones to have trouble are the people who are not guilty. This is going to turn into a waste of time all round.
Honestly the best thing that can happen is that someone produces software to generate infinite false positive pictures from the hash set database and that iMessaging them to random people becomes the new swatting. Overwhelming the review process may be the only way Apple ever see sense on this issue.
Using child abuse seems like a perfect excuse to justify increasing surveillance and reducing privacy across the entire Apple user base. After-all, how many people will argue against `protecting kids`? It’s really easy these days to be skeptical of someones true intentions.
No this is worse. The NeuralHash is going to be really easy to false negative.
I see NeuralHash as being able to claim the did something to protect kids when in reality the stuff is designed exactly not to do this. Think about it if apple only scans the images that NeuralHash pulls out with a proper method any one getting past the poor method is not going to get caught. This also will prevent apple from the public mess.
People forget there are such thing as AI face and body recognition. Using these would be harder to design but would be a more generic way of hunting down forbin images.
Yes using face and body recognition and age recognition would be searching the images for identifiable parts. Yes the auto censor programs like NudeNet are designed to hunt down images that contain porn parts and identify them and then finally censor them.
https://digitalcommons.uri.edu/cgi/viewcontent.cgi?article=2411&context=theses
Yes above is 2019 work of a proper made system to truly search. This AI system does not just hunt down known child expoit images also is hunting for unknown new child exploit images that could possible lead to a rescue.
These proper solutions don’t end up with the dog case of NeuralHash that is having match that looks nothing like the offensive image either. False positives in the proper solutions are porn images in all test cases so no case of taking a picture of a wall and possible getting flagged as the apple NeuralHash has.
Yes some people will be incorrect thinking that NeuralHash is like a proper Neural net solution for hunting down these images when its not.
Question – which is probably answered in some earlier post on this topic I haven’t read yet.. Does anyone actually think such pervs are actually going to put the photos in iCloud? How stupid can they be? (Perhaps a self-answering question)
I have no idea but would expect various experts to have a view on the various likelihoods. There’s going to be absuers who nevertake photos, those who do and they never go near the internet, those who may distribute, those who only download. People can and do make mistakes too. I have absolutely no idea whether the volume of detected or reported abuse is the tip of the iceberg or gathers the vast bulk of it.
In the UK a social worker of all people got caught when he accidentally sent images to work colleagues on a Facebook group.
I’ve read statements by the police complaining it’s a huge and growing problem and they don’t have the resources to deal with everything out there. The problem is the police say this about everything not to mention ignoring or de-prioritising domestic abuse and rape or the legal system tearing victims apart so they don’t want to go on with prosecutions.
By conceptualising a system which others may copy Apple’s arrogance is placing human rights activists and dissidents at risk. Those are the very people trying to change the systems in other countries which lead to all manner of horrors up to and including war and famine.