Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.
Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.
↫ Sid at 0xsid.com
…it’s “AI” isn’t it?
All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.
↫ Sid at 0xsid.com
It’s “AI”.
Yes, all that you need to do to gain control over big, massively popular Instagram accounts is ask Facebook’s “AI” to send the verification codes to whatever email address you desire. That’s it. There’s no other steps, no other checks, no other verification. And the worst part is that this isn’t even a hack; this is “AI” working entirely as intended.
And these tools are now coding the Linux kernel, LLVM, systemd, PulseAudio, rsync, your browser, and so much more. What could possibly go wrong?
To be fair, this is a case of “AI” being helpful, just to the wrong people.
I wouldn’t be “fair” about this. This is the most stupid design/setup ever. It should NOT be able to even do this, however hard you try. This is plain and simple an absolutely terrible implementation. Some marketer/product designer overrode or never asked for any kind of implementation/security review, and just YOLOed it and pushed it.
LLMs are pretty good at NLP interactions, but if you give it the keys to the kingdom, it’ll easily screw up. It should only have been able to say “send it to an email that was before any changes since the user claims it was hacked”. That’s not so hard, is it? Apparently it is….
Drizzt321,
I fully agree with this. If one is going to build an AI assistant with public access, there’s no way in hell I’d want it to have the “keys to the kingdom” and trust it to do the right thing….a terrible idea. It’s functionality really should be implemented through a human vetted API approved for public access.
The thing about this case is that we don’t actually know the facts. Did Instagram give the AI access to update user credentials at it’s own discretion? Or was the AI merely acting as a friendly interface to forward requests to the API as intended? This is a crucial distinction for assessing whether the AI has fault.
It was a joke. LLM chatbots have a (deserved) reputation of being unhelpful when it comes to real-life customer inquiries, to the point lots of people just shout “let me talk to a human” the moment they realize they are talking to an LLM chatbot.
Well, this is a case of an LLM chatbot being genuinely helpful. Just to the wrong people.
I hear the sarcasm in this comment. Made me chuckle 😀
Replace “AI” in Thom’s rant with “computers” and you probably reproduce a lot of the water-cooler talk in the 70’s and 80’s. Can you believe that they are going to let computers handle our money? The world is going to hell in a hand-basket.
I am not into the AI hype but I am not into the AI anti-hype either. My thinking has changed quite a lot on this and where I land right now is basically “computers do what you tell them to do, not what you want, and certainly not always what you need”. AI does not fundamentally change that.
We used to tell computers exactly what to do with machine language (assembly language). We do not do that anymore. Mostly we express the “semantics” of what we want in higher level programming languages. These go through compilers and interpreters to generate machine language. My old thinking was that compilers were deterministic and AI was not. Except neither of those things is as true as I thought. They are closer than I used to think. LLMs are much fancier of course but a bigger difference is that we give compilers much more detailed “prompts” about what we want (better semantics).
In both cases though, if they are not doing what we want, it is probably because they are doing what we told them to do. It is almost certainly that in this case.
If you have ever written a compiler, you know that the machine code it creates may implement different algorithms than the code you wrote. Entire variables, loops, and even functions can disappear. Loops may be rewritten. You get vector math and parallel processing where you did not ask for it. Even things like basic math can be converted to shifting binary bits instead. And what is kind of crazy is that, these days, it happens again at the assembly language level inside the silicon itself. Those instructions are further converted into micro-ops (microcode) which is what the microprocessor actually executes. And those micro-ops can be executed out of order, executed in parallel, or skipped completely. It is layers and layers of abstraction.
A program language like Python lowers the bar in terms of the knowledge you need to make a computer do things compared to having to write programs in machine language. An application created in a high-level language lowers the bar even further to make computers accessible even to people that cannot Python. But this means that the “users” can make even bigger and less well informed mistakes. AI takes it further in that it makes even more capability available to even less technical users. They are now free to make even bigger and even less well informed mistakes.
At some level “programming” is a management problem. You need to define tasks, delegate, and ensure that your minions are doing what you want. Even you need one part of the system to check the work of another to be sure that mistakes were not made. AI is even more of a management problem as it moves the human even further from doing the work but now even more responsible for making sure that the work being assigned is well defined, that the quality is being checked, that deployment is being done responsibly, and that change is being managed professionally.
Pointing at a poorly designed AI system and blaming the “AI” is like showing me a totally broken Excel spreadsheet and bragging that it is why you do all your math with pen and paper. The problem is the human in charge and your conclusion is wildly wrong. If you leave your AI with too much grey area, it is just like using a programming language with a lot of undefined behaviour or an employee with a poorly defined assignment. Tighten it up. That is your job as the manager.
LeFantome,
Same.
This is a poorly implemented feature for sure, but is there any evidence whatsoever that the process wasn’t created by humans? If the AI’s only role here was to assist users in interacting with account recovery forms, then it seems very unfair to blame the AI for the weak process. If it were customer support employees rather than AI submitting these requests, do you automatically go blame the employees who followed the process?
To be absolutely clear, I don’t wish to defend AI if it’s indeed true that AI created this account recovery process. That deserves to be faulted. However it’s becoming impossible to trust sources that are so hell bent on blaming AI regardless of what the facts are. I would genuinely like to know what the facts are.
1) Was AI used to create the authentication process?
2) Does the AI agent have any privileged access to user accounts without going through approved APIs?
3) Is there any evidence that AI didn’t follow it’s instructions?
Whether or not AI deserves blame depends on these facts and yet none of these facts are provided. I’m not trying to be factious, but it entirely possible if not probable the AI actually did it’s job correctly and that the fault for the weak process lies with humans who created the process. It’s not that AI can’t be responsible, but without the facts we’d be jumping to conclusions.
“My old thinking was that compilers were deterministic and AI was not.”
I’d argue that compilers are not deterministic. Compilers give you the same output for a given input using the same compiler options. That’s not true for any LLM.
r0ller,
I believe that you intended to say compilers are deterministic. Anyway I wanted to say that LLMs being non-deterministic is a choice and not a rule.
All computation units in a discrete computer are technically deterministic (absent hardware glitching). Non-deterministic output requires non-deterministic input sources. Random sources can be subtle and might be unintentionally overlooked, but absent a difference in input you’ll always have the same output.
When it comes to non-deterministic LLMs and content generating algorithms, this is because they were intentionally designed to have sources of randomness. But if you replace their random number generators with deterministic ones, you actually would get a deterministic LLM.
I always find that pondering deterministic behavior leads to some really deep questions about ourselves. Are we deterministic? It’s impossible to do (without simulating it), but given a 100% precise reproduction of a brain, will all instances output the exact same output for the exact same input? And what role does quantum uncertainty play in analog brains? How long would it take before brain states would start to noticeably diverge? These questions are tough to answer in physically analog domains, but it’s more straightforward in digital systems – even complex algorithms like LLMs are discrete systems simulating analog ones.
Here’s an anecdote:
About 8 months ago, my mom, who is 76-years-old, had a Facebook that was permanently disable for trademark infringement because Meta AI renamed her “Meta AI”. I don’t know what she was doing, but she clearly did not name herself Meta AI. She asked for help and I quickly identified the issue based on the emails Meta sent. So I sent an email to Facebook support stating the problem and they answered back that they could not re-enable the account without the trademark holder’s approval. Mind you, the trademark holder is Meta. I mention this to the support person and they say there’s nothing they can do. So she still has her account disabled, which I guess is good because Facebook is complete crap.
Haha, I’ve had instances of bad support from major brands, but that’s a doozy.. They should have known to escalate it. However I know from experience on the inside that escalating cases can be frowned upon – as in employees being told not to do it. The only hope your mom would have is if she had a social media post go viral. I know for a fact that some some companies like t-mobile employ people to scan social media and those employees are empowered to take action in ways that their own technical support can’t/won’t.
It comes down to acting differently in private and in public; they don’t want the public to see their dirty laundry. If you have clout in public, all of the excuses they throw at average joes simply go away and you get VIP service :-/
Speaking as a kernel maintainer, I think it’s worth pointing out that the Linux kernel, systemd and so on are large projects with extremely diverse viewpoints on the topics of how useful LLMs are, and indeed their ethical implications. In other words, we don’t all think it’s a good idea to let stochastic machines write our code. Balance is important, as it’s easy to feel like the sky is falling in and that every person involved with said projects supports this sort of thing, but I can promise you it isn’t the case.
The point is a reasonable one though, in terms of how long they continue to remain feasible with the noticeable negative effects slopcode contributions are having to general software quality around the world. Just wanted to make people aware it is a nuanced situation in a lot of cases.