Apple has announced impending changes to its operating systems that include new “protections for children” features in iCloud and iMessage. If you’ve spent any time following the Crypto Wars, you know what this means: Apple is planning to build a backdoor into its data storage system and its messaging system.
Child exploitation is a serious problem, and Apple isn’t the first tech company to bend its privacy-protective stance in an attempt to combat it. But that choice will come at a high price for overall user privacy. Apple can explain at length how its technical implementation will preserve privacy and security in its proposed backdoor, but at the end of the day, even a thoroughly documented, carefully thought-out, and narrowly-scoped backdoor is still a backdoor.
Basically, Apple is going to scan your iCloud photo library, and compare cryptographic hashes of your photos to known photos containing child pornography.
It’s hard to argue against this because it makes it seem as if you’re arguing against catching the sort of people that have such material. However, the issue with tools like this are not the ends – all of us are on the same side here – but the means. It’s more than obvious that this scanning is a gross invasion of privacy, but at the same time, you could easily argue that this is a bit of privacy we’d be willing to give up in order to aid in catching the worst elements of our society.
The real problems stem from the fact that tools like this are simply never going to be foolproof. Software is incredibly unreliable, and while a random application crashing won’t ruin your life, an algorithm wrongfully labeling you as a pedophile most definitely will. On top of unintended consequences, malicious intent could be a major problem here too – what if some asshole wants to ruin your life, and sends you compromised photos, or otherwise sneaks them onto your device? And with Apple’s long history of working very closely with the most horrid regimes in the world, imagine what governments can do with a tool like this?
On the ends that Apple is trying to get to here, we are all on the same side. The means to get there, however, need to be carefully considered.
Pretty horrifying.
People need to be clear on what image hashing means in this context. It’s not actually a cryptographic hash, almost exactly the opposite in fact. It’s a fingerprint designed to persist through common things that might happen to an image like cropping, re-encoding, etc. That means that unlike an actual hash functions, collisions not only happen but are a feature of the system, and that means false positives.
It’s rather like virus checking. I don’t think there’s anybody who has run anti-virus software that hasn’t encountered a false positive. Now imagine that the consequences of that weren’t that you had to navigate a few screens to rescue a known-good file from quarantine, but rather that the police execute a search warrant and take away all your electronics indefinitely.
I’m not in favour of corporate made law or corporations policing. There is also the problem of judgement without trial. Yet again I feel this is a none innovation done for marketting reasons. Myself I feel it more appropriate the security services and police and other regulated authorities such as social services and schools and other agencies handle this because they are accountable by law and their methods and decisions can be scrutinised. Once Apple has access to your photos what else are their servers up to they are not telling us about? Who scrutinises them? Who has access to data? What are the policies? How are staff trained? Just because American law treats your data as their data doesn’t mean Apple should exercise that and if Apple do this opens the door to an awful lot of other bad things with no oversight or scrutiny or redress.
Another problem is what if prosecuting agencies due to budget pressures become over reliant on this kind of thing as a source? The number of abusers with family albums full of smiling faces is not a short list. In fact I cannot think of a single prosecution involving abuse by a trusted person whether a family member or teacher or other person where a camera was involved and this is where the overwhelming majority of abuse happens.
Another problem is abused children not being believed or police not pursuing prosecution due to lack of evidence or courts utterly monstering witnessness, Now there is a lot of work which has been done to reverse this and a lot is ongoing. One example is policy which states the abused must be believed on the basis the allegation is true until an investigation proves otherwise. The biggest opposers of this kind of policy are far right libertarians. They claim it is a charter for liars and those seeking retribution. It is not. It is simply to force a proper investigation to confirm the basic truth of the matter rather than it being thrown in the bin before anyone bothers to look. Oddly, these self same far right libertarians allow abusers on their free speech platforms on “freedom of speech” principles while at the same time alleging very loudly that abusers are beyond the pale and would string them up if they got into power.
The proliferation of bad things on the internet is a concern but I have read of police prosecuting hundreds of abusers at a time so police do have a handle on it. The biggest concern I hear from the police isn’t their ability but more the lack of resources to track and investigate and prosecute everything out there. In theory catching the problem at source before it leaves the camera is a good idea but my original concerns still remain.
This is a much more compelling argument to me than the whole invasion of privacy schtick. I’m not entirely opposed to this kind of thing for the greater good, but I want very clear lines to be drawn and a lot more transparency than Apple has traditionally displayed. I actually think the technology here is pretty good and in the right circumstances this is even an ethically defensible idea, but not from Apple… I have no reason to trust Apple and I have no oversight over them to keep them in check. You can say “that is what the government is for” and my answer is before I can trust the government to do it I have to actually be able to trust the government…
Once they’ve demonstrated a technical capability and willingness to use this kind of fingerprinting technology to check data on someone’s phone for illegal content, the various copyright industry lobbies (eg, RIAA/MPAA) will be lining up to demand Apple use it to scan for “illegal” copyrighted material – basically Youtube’s Content ID on steroids, on every phone. In several jurisdictions demonstration of even theoretical ability to assist copyrightholders in preemtive enforcement has been used to obligate compliance with copyrightholders’ demands.
Imagine you send someone a photo that happens to include a TV with a frame of a film in the background, and then your Apple ID is terminated for copyright infringement and you receive an extortion demand for “settlement” from a film studio’s legal firm…
Ill play devil’s advocate on this. As it stands now, this hashing process is only applied to content that is created by the phone’s camera, it does not apply to content acquired from external sources. I suspect Apple will remain pretty adamant about keeping it that way. Yes, there is a slippery slope argument to be had, but I don’t get the impression that they are very interested in policing the internet for the RIAA/MPAA… I could be wrong though. Anyway, if they decide to go that route they are going to lose ALOT of customers right out the gate, that is pretty much a given. They might be that stupid, but I don’t think so.
Sure. That sounds like a great way to make customers happy… Some things probably wont happen just because its antithetical to doing business successfully…
Ill correct myself… I read someone else’s condense explanation of CSAM before reading the write paper so my bad. The white paper does not specify that this is limited to photo’s created by the users’ camera, only that it is applied to images stored in iCloud. I have no interest in playing Devil’s Advocate anymore, that is a deal breaker for me.
If they wanted this to be limited to the stated purpose it would be limited to photos coming directly off the camera (i.e. this would be a way to find the source of later identified photos worth investigating. As it is described in the white paper, it goes way beyond that. This would facilitate going after people in position of copies of said photos. That is not what I signed up for. I have no interest in policing people’s sick fetishes, no matter how sick they are.
Boo.
The technology they’re using simply can not work for that. It creates a fingerprint (“hash”) of a picture; then checks if the fingerprint matches any of the fingerprints corresponding to “disallowed pictures” in Apple’s collection. Anything that is not in Apple’s collection (e.g. anything from the camera) shouldn’t match.
The end results are:
a) pedophiles will stop sharing old pictures on their iPhones/iCloud (because they’ll be detected/caught)
b) the value of new pictures (that won’t be detected) will increase, causing an increase in people creating new pictures for profit.
c) pedophiles will switch to sharing pictures via. other methods
d) there will be some false positives (you take a photo of a birthday cake with no people in it at all, and suddenly you’re flagged as a pedophile).
e) It completely ignores motive. If collecting or sharing child pornography makes you a pedophile; then “Apple is collecting child pornography (to make their fingerprinting work)” makes Apple pedophiles.
Sure it can. This is the scenario I am talking about (and the one I think is at least somewhat ethically defensible):
1. User take photo with camera of illegal act. The photo is hashed (because it came from user’s camera) and it uploaded to iCloud. Only photos taken on camera are ever hashed when uploaded, if they come from any other source they are ignored.
2. A copy of the photo is later discovered on the internet by police/investigators and is flagged as serious enough to be worth investigating its origins. Apple is made aware of this photo by law enforcement.
3. This publicly sourced photo is now hashed by Apple and added to their CASM list. Only then would their detection algorithms potentially find it on someones device.
The point is if a photo with a matching hash is found on a device and its verified that it is the same photo, its not just a copy of the photo, it was taken on that device. If implemented they way i described, it in fact can only be detected on that single device…
That seems reasonable to me, or at least much more defensible, because it is going after the source of the photo, not someone who simply acquired it from the internet. Like I said, I have zero interest in policing people’s fetishes, no matter how sick I might consider them.
It does not matter though I guess, because it does not work this way afaict.
You’re right – it can work (in theory), if Apple re-checks the fingerprint for every file whenever a new photo is added to CASM.
I’m not sure how practical that is though. For a crude “pulled out my butt” estimate; if there’s 1 billion iPhone users who take an average of 1 photo per day each; then, after ~10 years, they’d have to re-check the fingerprint for about 3 trillion files every time a photo is added to CSAM (and deal with all the false positives, and store people’s original files and not just the fingerprints/hashes so they can deal with false positives, even after the file/s are deleted by the original uploader so that they can determine who the original uploader was).
Just to clarify – I wasn’t saying Apple would voluntarily enact copyright enforcement, I was suggesting they may be forced to, having demonstrated that they can – from a technical standpoint the difference between recognizing this kind of material and recognizing known copyrighted works is basically nothing (and in fact seems to mirror measures the EU was trying to push a few years back around copyrighted works databases)
And yeah – they’re scanning stuff on your phone, currently only when it’s queued for transfer to iCloud/iMessage but that’s pretty trivial to extend to all stored content (and it will be, if this goes ahead).
And of course if Apple goes ahead and does this, Google will be pressured (by lawmakers and activists) to do it too. Likely Microsoft eventually as well.
I see your point and agree there is the potential for this to be abused.
To be fair though, your point “…an algorithm wrongfully labeling you as a pedophile most definitely will” is inaccurate as Apple have stated that anything flagged will be double checked by a human first, before contacting authorities.
I am by no means an expert in this supject area but I understand there is a range of images from the graphic to borderline to innocuous. Some are obviously abuse images. Others may be artistic or family pictures. There are others which look wholly innocent but when cropped or used as part of a composite become something else. It naturally follows beyond a certain point someone with expertise and access to more knowledge of the situation whether gathered beforehand or afterwards needs to be involved. This will be law enforcement unless Apple wish to set themselves up as vigilante’s.
To complicate things more parents have posted videos online of young children (and I daresay photos too) of family or life achievement events. The content has been seperately tagged by abusers who with commentary re-interpret or give timestamps so the wholly innocent and lawful content within their community becomes abuse content.
In order to “check” it, Apple themselves need to copy, transmit and store the image, and an employee or agent of Apple has to view it.
If exploitative material, that’s illegal in itself and frankly abusive of the employee involved.
If not explotative material, who the hell are Apple to get third parties to access your private material?
And then of course – private actors are not police, not able to access surrounding context, and are not going to be in a position to make a legal evaluation on whether a given item is legal in a given jurisdiction (or, say for instance in the case of minor users’ self-created material, if it’s something that should be handled with with more tact and discretion than the entire FBI or equivalent breaking down doors)
Wow! This opens up a very terrible avenue for scamming online accounts.
Remember the story a few days ago in the new cycle: https://www.vice.com/en/article/k78kmv/instagram-ban-restore-service-scam . Scammer are charging $60 to have Instagram account banned for 24 hours by issuing fake suicide notices.
Take that, and add this new system, and you get a permanent lock out of icloud. What would happen, if someone deliberately sends you an image that is hit by this hash, but actually legal (to protect themselves)?
Will the hash flag your account? Yes, most definitely.
Will your account be put on probation until a manual check by human moderators? Yes, most likely.
Are humans prone to errors? Yes, some of the time.
Will this cause false positives for people being locked out of their account? Yes, one in a million, but there are billions of users.
Will those account be notified for the reason of this ban? No, most likely not.
–What would happen, if someone deliberately sends you an image that is hit by this hash, but actually legal (to protect themselves)?-
https://rentafounder.com/the-problem-with-perceptual-hashes/
Problem here it does not have to be deliberate attempt to harm you. The match demoed by the person work with with related forms of ai helped hashing.
Think for one min if someone creates a meme image from existing image in the process add some text it is theory possible that you will go from a not matching image to a matching image. They proceeed to send that and and people start forwarding on because it looks cool.
The type of hashing apple talking about using is not that predictable. The kind of hash that used some anti-virus use as part of their heretics that is like the apple hash. that we know false positives even that the fact on paper some of the hash-ANN like has for antivirus have less than 1 in googol error rate but they still at times spit out completely wrong false postivies. Yes 10 to 100. Trillion is either 10 to 12 or 10 to 18. Turns out you artificial intelligence hashs we cannot mathematically calculate error rate in a general way. Each artificial intelligence hashs that is generated has it own unique error rate. The number of images to be searched for is always increasing. So there are always new hashes being created.
The reality is in even if apple is tell the truth 1 in a trillion error rate with a ai hash would be a average this could have a range of errors from 1 to 1 to 1 in 2 trillion every time you create a hash and still have a 1 in a trillion error rate. This is the maths nightmare of a ai based hash you really do need the range of possible error values. Remember each new AI hash turns out to be a game of random. Some of the hashes will be good and low error rate some will be very high error rate.