AWS: Not even once. This prominent Ruby developer lost his entire test environment – which, ironically, was pivotal to AWS’ own infrastructure – because of a rogue team within AWS itself that apparently answers to no one and worked hard to cover up a dumb mistake.
On July 23, 2025, AWS deleted my 10-year-old account and every byte of data I had stored with them. No warning. No grace period. No recovery options. Just complete digital annihilation.
This is the story of a catastrophic internal mistake at AWS MENA, a 20-day support nightmare where I couldn’t get a straight answer to “Does my data still exist?”, and what it reveals about trusting cloud providers with your data.
↫ Abdelkader Boudih
Nightmare scenario doesn’t even begin to describe what happened here.
AWS is deeply in the wrong of course, but how does a developer not understand that having “backups” in the same place as your data is bad practice? That’s not redundancy. The point isn’t having data duplication, the point is avoiding having any single point of failure.
This guy seems to have learned that AWS isn’t trustworthy, but that’s the wrong lesson.
This was a test environment and not production; his production data and anything not related to his specific Ruby on AWS test area was backed up separately and was not affected. Besides, it wasn’t just the data itself that he lost but the entire account as a whole. He had it all backed up to his laptop but he has to start from scratch with a new provider and rebuild his entire open source testbed on a different platform to be able to continue his work on Ruby projects, many of which were in use by AWS themselves where their devs depended on him for (free) support.
The point being, despite AWS’ own assurances of a 90 day grace period where his account and data are supposedly frozen in time and able to be recovered, this team within AWS fucked up, accidentally wiped his (and several other developers’) entire accounts by way of a typo in their own shitty “test in prod” environment, and instead of owning up to it they tried to cover it up. He only found out because one of them was quitting and had enough of a heart to spill the beans on what actually happened.
This isn’t about having backups. This is about how AWS’ “move fast and break things” test-in-prod approach on the back end, combined with apparent zero accountability when they do indeed break things, is a recipe for disaster for anyone daring enough to use their services.
Morgan,
That’s a fair point that in this particular case the customer had external backups (good on them). I’ve heard about multiple cases over the years where AWS held everything and they did not have backups.
https://www.indianweb2.com/2025/02/construction-firm-claims-rs-150-crore.html
> but how does a developer not understand that having “backups” in the same place as your data is bad practice?
It is not the same *place* that was the problem; aws manages multiple redundancies (including cold storage vaults) at a level no single dev could hope to match.
The problem was using the same *service* and that service refusing to restore said backups.
Mote,
I agree. You can have your data span across different AWS regions, but the lack of geographic redundancy isn’t the cause of this particular problem. It’s placing all your eggs in one basket. If the data is critical, then there’s simply no legitimate excuse, it’s inherently irresponsible to keep the data with a single provider. That said, this is the way a lot of businesses and even governments are run today. Most customers don’t experience problems most of the time, which gives many a false sense of security, but errors/accidents/breaches/employees having a bad day/etc do happen. Even if the odds are one in a million, it could still be devastating if that happens to be you and you’ve kept all your eggs in one basket .
“Does my data still exist?”
If it was all in the cloud, it never was “your” data.
Hmm….
It is at least “someone else’s computer”
Whether the data ownership is there can be debated. But the custodianship is not in your hands anymore.
This is not the first time a well-known online services provider loses stuff and tries to gaslight users about it. Google Drive lost some users’ files once and they also gaslight users and denied everything.
You see, you don’t have to admit to data loss if you deny the data loss happened, and for companies offering “cloud” services, marketing their online services as free of data loss is essential to their marketing.
kurkosdr,
Their best “offer” would be refunding your money. Unless you already have a multi-million dollar special agreement with them, they will be unlikely to care too much about it (except for reputational purposes).
they can’t ever admit, own it, nor refund anything, because thus they’d open up to becoming all dead and deep buried in ensuing civil litigation(s), is why they can only ever play dumb & gaslight;
this’ll get exponentially more pronounced the more they selfsaddle with ais, prone to exactly the type of elusive typos in question.
Refunding money would be an admission that data loss happened and that it was AWS’s fault, AWS wants to avoid that because they market their service as a service where no data loss happens, ever, hence the gaslighting.
I feel the larger issue — and what makes me absolutely in no way ever want to use their service — is that this act of negligence was performed by a small team within AWS that seemingly has no accountability and can get away with nuking an account in a way that defies AWS’ own guarantees about redundancy.
According to AWS’ marketing and SLAs, unless you are found guilty of defrauding them, any account issue that leads to termination or deletion of your account can be reversed because it’s never really deleted, there’s always a backup somewhere in their system because they have redundancies for their redundancies. Your account is not supposed to really be deleted, even if you delete it yourself on purpose, until 90 days have passed. That is their guarantee. However, this rogue team has the ability to completely obliterate your account with no chance of restoring from a backup, just by leaving out one “-” in a script. It proves that there actually is no guaranteed redundancy with AWS if the wrong knob is tweaked. Letting this knowledge go public would undermine every corporate and private customer relationship they have, and so this team is keeping their mouth shut and trying to, as you put it, gaslight the author into just giving up.
I hope this story gets some independent verification and blows up in the tech news media beyond this site and Lemmy (where I saw his blog post).
Morgan,
What happens in these kind of situations is that the organizations update their policies so that this never happens.
This is common in mission critical systems. Take The International Air Transport Association (IATA). They take “black boxes” very seriously. The reason is they want to figure out why an accident happened even if that was 10 years ago. And they update the rules like “no high heel shoes while evacuation” since at least in one instance it caused a catastrophe.
I can talk about Google, as I know their practices. There used to be a “blameless post mortem”. If you caused a minor or major accident you were supposed to write one of these.
These not only stay as part of the company knowhow, they are also used to implement new tests and policies to ensure the same exact mistake does not happen again.
Back to AWS.
I’m pretty sure they have started looking into preventing this from happening again.
It of course won’t make this particular customer whole. But the next one is much less likely to be affected.
@sukru:
Having worked in local government for a large portion of my adulthood, I’m aware of incidents like this causing quiet policy updates. I’ve written said policy updates myself, and as long as no one was harmed the policy update generally isn’t public knowledge (in my former field anyway where we weren’t required to disclose minor incidents to the public).
My issue with AWS in this instance is that harm was done, and unless that rogue team is fully audited (and outright abolished if they don’t pass audit), no amount of policy addendums will prevent another incident like this. I realize all we have to go by is the author’s side of things, but I guarantee his informant didn’t disclose everything wrong with the MENA team. Honestly they should be fired and replaced with a properly vetted team that has been made aware of the consequences of this level of fuck up, AND the policy/procedure docs should be updated alongside this action to enforce future compliance with said policies.
I realize I sound like a soulless suit saying all of that but honestly I’m 100% on the side of the author in this case, and my heart goes out to him. I’m glad at my current job I don’t have to deal with shady teams doing things under management’s nose and getting away with technological murder. I’m the one responsible if something goes wrong with our network, and I take steps to prevent bad things from happening.
Morgan,
I hope they don’t fire the person who did this, … unless it was on purpose (unlikely) or they failed to follow procedure.
I have seen many mistakes done in large corporations, I had also done my fair share of them (worth X millions of dollars of loss). However at the end of the day learning from these mistakes is part of the job.
Yet… the other part… trying to hide responsibility is bad. And if this is intentional… then yes there should be some heads rolling.
@sukru:
That’s what I think happened based on the author’s side of the story and that’s why I feel they should be fired. Mistakes happen, but covering up a mistake by gaslighting the victim cannot be tolerated in any workplace.
Morgan,
That is fair. If that is the case, those kind of people have no place in a modern workspace.
Feel for the guy, but not having offsite (ie. off AWS) full backups is a pretty big fail.
Just in case anyone didn’t make it all the way to the bottom of the OP, please note that there’s a follow-up story:
https://www.seuros.com/blog/aws-restored-account-plot-twist/
TL;DR? He did indeed get his account back.
Kuraegomon,
Thanks for the update, that is very interesting!
Makes me wonder “why” they refused to restore the data and what changed.
The author says…
However this seems somewhat contradictory to me. Without the widespread public coverage, it does not seem like any more progress was going to be made. And the employee who reached out seems to indicate that the social media coverage and bad PR may indeed have been responsible for greasing the wheel here…
He goes on to explain how his previous AWS support interactions were filled with misinformation. The question becomes why?
1) Never rule out incompetence/bad training
2) Lies/laziness (I really hope not, but maybe….)
3) Support staff are genuinely out of the loop. Maybe they did their jobs correctly, but their tools have been lying to them as well such that an engineer was required to get the data back.
This is completely tangential but for privacy purposes how much faith should we be putting in statements made by companies swearing our data is deleted? In this case the user wanted the data back, so all is well that ends well. What if the circumstances were different though and the user actually expected the data to be gone. A lot of private stuff is held on these “cloud” services and if it turns out they’ve been lying about the recover-ability of our data then it opens up a lot of serious questions about trust. They say it’s deleted, but is it actually?