“Hopefully you never had to restore your own system from a compromise and you will not have to do this in the future. Working on several projects to restore a compromised Linux system for various clients, I have developed a set of rules that others might find useful in similar situations. The type of hacks encountered can be very variate and you might see very different ones than the one I will present, or I have seen live, but even so, this rules might be used as a starting point to develop your own recovery plan.”
are you joking, linux can get hacked ?
no, seriously, wait to see the number of posts from Windows users saying;
“Told you linux is not secure”
“I am better off with Windows ™”
“My Windows is more secure than linux”
“Thats the problem with open source, everyone can see the code”
and so on and so forth………………..
only if windows user == windows zealot…
A system can’t be completely secure as long as there’s a way in. If there’s a way in, there’s a way in for hackers to find.
Security is about making that really, really hard.
and Windows’ lack of security has to do with MS making it really, really hard to make it really, really hard.
(At least for home users who don’t have a decent router+firewall
)
I’m pretty sure this is a myth. If there was a system open to the internet that was just set to respond to any packet by relaying it back, there’s probably not a way to hack it unless it’s IP implementation is severely broken.
I believe that you can make a system unhackable it’s just really really hard.
You forget, I said there had to be a legitimate way in.
Your hypothetical system could be cracked via physical access.
This would be an outrageously extreme example: An unconnected computer locked in a military base, with a 12-step authentication process including a physical dongle, of which the combinations are only known to Major General Heironymous Pingletron.
In all practical terms, that would be impossible to hack. And yet if there’s a way for Major General Heironymous Pingletron to access the system, then there is a way in someone could theoretically find. Maybe if they were James Bond.
Obviously in the real world such levels of security would be only for those beyond paranoid, and one could design a much more reasonably impenetrable system.
Assuming you can determine the start date of the compromise (not always easy), one option might be to wipe the OS/data, re-install the OS and recover data from tape back to the day before the attack. You’d still want to snapshot everything that was on the original system for investigation and possible recovery (because you may need to bring the data back up-to-date).
This is most effective on a server that’s just Web serving and stores its DB info on a separate server (because the Web server won’t change its filestore very often, Web logs excepted). Of course, your DB server shouldn’t even be visible to the outside world, so it’s far more likely the Web server will be hit rather than the DB server.
…To the Debian developers.
Twice.
AFAIK the “root” cause of both incidents was weak passwords (followed by exploits). Kinda OS independent. Community projects are more susceptible in this area.
…To windows developers as well…atleast Linux servers can be restored… and they dont need to format and os dont need to reinstall again like…
“Assuming you can determine the start date of the compromise (not always easy), one option might be to wipe the OS/data,”
This is actually *always* the correct, and only, course of action. If the system has been compromised the only way to get it back to a clean state is to wipe it out and restore valuable data from backup.
Trying to find out what files have been tampered with and trying to just remove and replace compromised files is dangerous. Since the machine has been owned you cant trust anything it is telling you, be it timestamps of files or the output of ps.
While this article has some good points, such as disconnecting the system and doing a forensics analysis, it is a fatal flaw not to wipe the system out and start from scratch.
Eh, it depends. When the original internet worm hit, we were able to grab the source code and knew exactly what it was doing. There was no need to wipe and restore those systems.
A lot of exploits are naive and can be disenfected, if you’re properly set up to do so.
In a multiserver environment, there are ways to externally verify the integrity of system files. On BSD, a handy technique involves running mtree periodically and saving the result off-system.
If you are going to periodically audit a system it’s a very good idea to keep the audit logs on a separate system in a read-only format. Burning them to write-once CDS works.
If you can hook the kernel, Tripwire won’t help you. That means you can mess with the file system drivers, and make the replaced files appear as they were, replete with hashes.
Think of how people get around this with Alternate File Streams in Windows.
Even if this is one of those simple PerlBot or PHPBot hacks, you wipe and reinstall.
If you want to investigate, get yourself a write-blocker and/or mount the filesystem from a live CD such as FIRE, off the network. Knoppix and its variants also work quite well.
For Windows, a Write Blocker and BartPE (the UBCD variant) work really well.
When you rebuild, you make double sure that you are only running what you need, with only the permissions it needs. Tools like GRSecurity, PAX, and SELinux are really good here.
Unfortunately, many LAMP/WAMP apps are guilty of excessive permissions on the app and database sides.
When you look at what you’re running, always think of what is the absolute minimum you need to get it to run.
Also make sure that you’re sanitizing all user input. Additionally, make sure you use parameterized database queries, and none of this dynamic SQL garbage
.
Password complexity is also a given, as is the use of SSH Keys where appropriate.
The best way to recover from a hack after reinstalling is to make sure that you are running the most up to date software, and that you’ve reviewed your app code to fix as many holes as possible. You also want to make sure that all your apps run with as few permissions as possible, and that you’ve taken all the precautions you can.
I see too many apps these days written in PHP that are easily exploited to turn machines into bots. Hopefully the author can add these suggestions to their toolkit.
Sorry, there’s no way you can trust a hacked box. Reinstall and recover from a trusted backup is the only way to get around such a situation.
`Sorry, there’s no way you can trust a hacked box. Reinstall and recover from a trusted backup is the only way to get around such a situation.`
I completely agree. Especially at the data center level and if its a `proprietary` operating system. Having the registry cracked is a complete waste of time to repair or rebuild.
Linux on the other hand is actually a pleasure to harden, monitor, and to repair.
Edited 2006-07-18 11:55
Sorry, there’s no way you can trust a hacked box. Reinstall and recover from a trusted backup is the only way to get around such a situation.
That’s the usual knee jerk, but like all knee jerks, it’s simply wrong.
If I have the source for the exploit, and from that can determine what the exploit does, I can decontaminate the box without a reinstall and recover. We did that with the original internet worm and it worked fine.
If I am running a system for which the root file system is read-only, I can safely avoid the reinstall, since it will be unnecessary. Mounting your root fs from a read-only optical media, for instance, pretty much guarantees that it’s not going to have been modified. (In my world, read ‘rom part’ for read-only media)
If I have a properly installed system with audited base configuration and I can externally verify the audit, I don’t need to reinstall, but I may need to replace configuration files.
I will grant you that few sites set up their installations so that the above can be true, but those that do don’t need to wipe and reinstall on every exploit.
Besides, if you wipe and reinstall, how do you know your backups weren’t compromised before the exploit was detected?
I’m no system administrator but I see a red flag here. Yes, if you wipe the OS and restore the data you end up having a clean system. No problem there. But have you noticed that by doing that you just installed a system which the cracker had no problem cracking? What stops him from doing harm?
That’s exaclty what the article touches. The objective of the approach stated there wasn’t to simply clean the system. It’s objective is to clean the system, learn why it got compromised and avoid it being again compromised a few minutes later, with possibly further problems and consequences.
What you are advocating here is a endless cycle of compromising/cleaning/installing. You do not pay attention to the fact that you are treating only the symptom and not curing the disease, which makes me pray that you are also not a system administrator.
> What you are advocating here is a endless cycle of compromising/cleaning/installing.
Er, you clearly didn’t read all of my original posting then. I said: take a snapshot of the system for investigation (you *do* need to find out how they got in after all and close the hole – geez!), wipe the OS and restore data from just before the compromise (you may then want to consider how much of the data you can safely restore from tape or the snapshot from the compromise date onwards).
Clearly, the original article’s methods then kick in – find out what they hacked by examining the snapshot, fix it (e.g. remove or patch software/data that’s compromised) and only then put it back on the Net into full service. Nowhere in my posting did I say that the article author’s methods were wrong and should be ignored, but I was just suggesting another way to recover the system that could be quicker (because once compromised, the hacker could have added loads of further local-only hacks and covered their tracks well enough that you could miss them and indeed go through the endless cycle of restore/fix that you accused my posting of encouraging!).
Yes, there is a way but it takes a lot of trouble and the person doing it must really know what he is doing.
But if you had read the article you would notice that what the author wasn’t advocating putting online a hacked box. What the author states is that to recover a compromised system, doing a clean install isn’t enough. The system administrator needs to know what was the point of entry, shut it down, clean the system and only then put the system online but always monitoring it for further signs of mischief. If you had read the article you would noticed that the vast majority of the tips were relating to system inspection.
So, as it is easy to see, cleaning a system and reinstalling obviously isn’t enough. The system administrator needs to know what went wrong and make sure that it doesn’t happen again. If the system administrator only cleans the system and puts it online, he will find himself in the exact situation he was before and probably in a mindless cycle of compromise/clean/reinstall
With all respect towards your openion, I’ve been a sysadm for 6+ years. I did not talk about finding out what went wrong or putting the system offline/online, that’s trivial, sure you do it.
The sad part is that you can never trust a compromised system anymore and you absolutely have to do a clean install to restore your service in the proper way. The article clearly seconds my openion anyway stating that
“..you will hear that you have to fully reinstall the system and start fresh because it will be very hard to remove all the hidden files the attacker has placed on the system. This is completely true and if you can afford to do this then you should do it.”
My point is that there’s no second option on maintaining a mission-critical server. You gotta do everything what’s mentioned in the article AND do the fresh install to replace the service.
[Edit: fixed typo]
Edited 2006-07-18 12:29
My point is that there’s no second option on maintaining a mission-critical server. You gotta do everything what’s mentioned in the article AND do the fresh install to replace the service.
If your planning did not include the eventuality of an exploit, you are correct. However, if your planning did include the eventuality of an exploit, there are ways to avoid the need for a fresh install in many cases.
The time honored way of doing this is to separate your base system from your volatile data and install your base system on media that requires physical intervention to make writable. A system that can’t be written can’t be compromised.
If you don’t have such media, then using tools like mtree coupled with off-system storage of the audit and the ability to verify from a live cd will allow you to determine in some circumstances that you haven’t been compromised. In that case, you don’t need the reinstall.
If you use something like the FreeBSD jail (or similar chrooted environments) to contain services, you can guarentee that a compromise of a service doesn’t not result in the compromise of the entire system, so you only have to reinstall the compromised subsystem.
And in the end, you’re stuck wondering whether or not your backups were compromised, anyway, since, at your level of paranoia, there’s no way to know when the exploit occured.
“Eh, it depends. When the original internet worm hit, we were able to grab the source code and knew exactly what it was doing. There was no need to wipe and restore those systems.”
How would you know that it hasn’t been used to install additional exploits and/or rootkits? Granted, the Morris worm didn’t but it’s not 1988 anymore.
Plus, the Morris worm isn’t exactly the same as being “hacked”.
If someone unauthorized has had root access to your systems all bets are off.
“In a multiserver environment, there are ways to externally verify the integrity of system files. On BSD, a handy technique involves running mtree periodically and saving the result off-system.”
And what if the intruder installs a modified mtree binary that reports everything as normal when it isn’t? How do you save it “off-system”? What if that process got corrupted?
In the end, sure, maybe you can recover a system but it involves a lot of timeconsuming work and if you already have backups of your important data it’s safer and probably faster to just to reinstall the OS and restore your data.
“If you are going to periodically audit a system it’s a very good idea to keep the audit logs on a separate system in a read-only format. Burning them to write-once CDS works.”
Agreed, that is a very good idea and mtree is a very nice tool.
How would you know that it hasn’t been used to install additional exploits and/or rootkits? Granted, the Morris worm didn’t but it’s not 1988 anymore.
In the case of Morris, we reverse engineered it, and it was very obvious. It’s not ’88 anymore, but there are a lot of ’88 level naive exploits floating around script kiddie land.
If someone unauthorized has had root access to your systems all bets are off.
What’s off is any assumption you can make about the integrity of the system. The next step is to verify what you can about the integrity.
And what if the intruder installs a modified mtree binary that reports everything as normal when it isn’t? How do you save it “off-system”? What if that process got corrupted?
You run mtree from a live-cd, using your off-system copy to verify the on-system tree. Since, at recovery time, you’re verifying data using a completely separate second system that is on read-only media, you trust it. That’s no more trust than you give to whatever tools you use to wipe and reinstall.
In the end, sure, maybe you can recover a system but it involves a lot of timeconsuming work and if you already have backups of your important data it’s safer and probably faster to just to reinstall the OS and restore your data.
This is probably a good place to mention sandboxes and jails…
The trick here is containment. If you set up your systems so as to make containment easy, you can find yourself in a situation where it’s easier and faster to decontaminate than to reinstall the entire system.
But yes, there are times when it’s simpler and faster to wipe and reinstall.
Is that Marius does not consider the possibility of the company that owns the hacked server bringing in law enforcement. The server(s) would be considered evidence against the malicious user(s) and any attempt to mess with the machine could destroy evidence. About the only thing you can do if you are going to call law enforcement is to unplug the machine from the network and wait.
Once law enforcement personnel glean what they need from the machine(s), then you can make your own analysis and rebuild.
You could simply put all your executables and configuration on read-only media — even mount them from another machine that’s isolated and exports
things read only. There’s no rational reason to load the kernel and kernel modules from read/write media, and little reason to do so with libraries either (unless you are constantly changing them).
That’s right. One of the reasons that /opt and /var were introduced into the Unix filesystem namespace and a variety of things were reorganized was to make it easier to mount the base system on read-only media.