Earlier this year, I released a proof-of-concept project called “EmbedExeLnk” – this tool would generate a Windows link (.lnk) file containing an embedded EXE payload. I have taken this concept further and created a tool that generates a Windows registry (.reg) file containing an EXE payload.
Nifty.
Smart. I didn’t realize this was even possible!
Windows registry has been broken for so many years. Just throw it out, i guess it made sense when you had 8mb ram. I now sit at 512gb ram and still can not get the grip of the registry. It is the main source of all infections in the US according to several posts from the NSA (that entire operation is dubious at best)
Why cant microsoft just remove it now in the age of ssd’s and processors faster than a 486?
The speed difference was at one point a valid claim, but as SGI showed: a file on a XFS file system can be reached faster than a binary call to the windows registry on a ntfs system. THE supposed benefit was called into question.
Phoronix has some biased benchmarks (they favout ext, as they see it as freer) but xfs still beat all of them in almost all use cases.
I’d rather see a new OS written from the ground up for `modern` technologies rather than the continued frankenstein-ation of Windows, bolting stuff over jerry-rigged stuff over bolted stuff. Building a nice house on a bad foundation doesn’t change having a bad foundation. Eventually It will need either a lot of bulk and resources to keep it standing, or it will be better to demo and rebuild using what you’ve learned.
I don’t like the registry either…I’d prefer a proper database. Building it around COM components and GUI ID indirection was awful. Surrogate keys are obviously needed under the hood, but it should be properly abstracted and properly enforced. Just having a mishmash of database concepts while trusting applications to manage it and telling people not to touch it is a non-solution. The registry is so much more difficult to use than a proper database. SQL would not only be more familiar, but much more powerful too.
I’m not a huge proponent of unstructured text config files, but at least on unix we’ve got a wealth of tools to manipulate them. The registry seems to be the worst of all worlds IMHO.
Yes, using the registry was the “right tool for the wrong job”.
I think Raymond Chen would summarize the history much better than I can:
https://devblogs.microsoft.com/oldnewthing/20110907-00/?p=9713
They had to switch from INI files to something modern. And they had the Registry database code, albeit in primitive form. And since they would not be able to fit a real database in 4MB of RAM which was standard at the time, they went with it.
With the benefit of the hindsight, it was not the best decision.
Interesting article, but you seem to be willingly misinterpreting the point. It openly says throwing a full SQL database at the problem would be overkill, and most developers would rampantly abuse it if they did. As a quick lookup for things like previous X/Y window coordinates or what program to open files with, it makes it seem the registry’s flat database style is a good choice. Storing these things in an ever growing, endless amount of config files and hoping nothing corrupts them will eventually require manual editing of the config files to fix things. Meanwhile far less than 1% of Windows users have ever needed to edit registry files for over ten years. It is the right tool for the job, a simple database for quick lookup of settings, and its atomic nature makes it much more resilient than config files. Of course if they actually made it a full relational database, most of the people here would immediately call it bloat.
dark2,
Why do you assume there’s more bloat in text files than in the registry?
While we don’t call it such, a file system is also a database in much the same way the registry is. It’s all just sectors on disk. The difference is a matter of APIs and organization. Personally I really like working with structured data over text data, but despite this I hate working with the registry. It’s clunky, offers none of the benefits of text files, and none of the powers of typical databases. On top of this it’s badly organized. There are just no winning qualities IMHO.
I’m not married to “SQL”, but it does have the benefit of being very familiar and powerful. Maybe it’s overkill, but for programming/automation/configuration it’s easy to see the appeal over both the registry and text files. Doesn’t have to be a full blown MS SQL server, lightweight implementations are possible.
friedchicken,
The problem with Windows is also their greatest asset. We have discussed this before, but having arguably the best backwards compatibility of any OS on Earth, is a major selling point.
I am not sure how they can fix this. At one point (Windows 7?) they tried including a whole Windows XP virtual machine inside, but even that had many issues and had to give up.
If, one day, we have modern, containerized OS, then we might actually see a clean Windows.
sukru,
Yeah, objectively bad implementations often have an underlying reason for keeping them around and backwards compatibility certainly fits the bill. While I agree with friedchicken’s sentiments, there’s also the good enough principal which means they may never be fixed.
With so much of the world running on legacy software & backwards compatibility needs, it practically guarantees Microsoft will not pull the trigger on a major clean-up, modernization or re-designing that would force their own customers into the same for their own (custom) software, at great cost to everyone. As you pointed out, the `good enough principal` is a real thing and probably the safest bet in our expectation of the future.
I’m not sure how merely loading a file (and not bothering to parse or search its contents to find anything in it) is comparable to actually finding something. It’s like saying “putting water in the kettle and turning it on is faster than actually making a cup of coffee, so people should just drink boiling water directly from the kettle”.
I also don’t see how having 100+ separate pieces of shrapnel (an individual hacky custom configuration file with its own format, its own tools, etc; for each different piece of software) strewn all over the file system could help any security issue. More likely is that it’d create 100+ different security issues instead of one (while reducing disk cache efficiency, increasing file path lookup overheads, and preventing people from easily backing up all their settings).
Brendan,
I’ve mentioned that I’d prefer a structured database to both windows and unix. That said though do you have any sources for your claims that the registry is more efficient? I mean it’s easy to assume, but actual benchmarks may show differently. My gut tells me that reading a few KB of a text file might actually be more efficient than using the registry API to enumerate the same information in the registry. The difference could be between a few syscalls on linux and hundreds for windows. Also, caching on linux is quite effective. So you pose a really interesting question, but I don’t think it can be properly answered with a handwavy argument.
In my own experience with this as a windows admin, I absolutely hated backing up and restoring the registry. I mean unless it’s all or nothing, it can be a nightmare to restore only the apps you want to restore like on a new machine. Merely trying to identify what you need in the registry is miserable. Sometimes as IT techs we would even recommended reinstalling windows from scratch just to clean out the useless cruft…It was such a common problem I even remember there being 3rd party utilities that tried to manage this registry junk. I eventually gave up backing up applications on windows all together preferring to reinstall and re-configuring because that was just easier than trying to mess with the registry at all and a fresh install cleans out most of the junk. YMMV.
While it’s fine that you’re not a fan of linux, I don’t think we can genuinely make the case that identifying/copying/backing up/restoring configuration files is harder than the registry. I do it all the time especially on new computers, after installing an app from a package manager I simply copy over the config dir/file and 99% of the time it just works.
I didn’t claim that (Microsoft’s implementation of) a registry is more efficient than loading a file (and ignoring a huge amount of work that would be necessary to make the comparison comparable); I only claimed that the comparison seems extremely dodgy.
I’d be happy to make the claim that a registry should be more efficient than having to parse through plain text in a sequential fashion (due to the absence of any indexing, etc) while dealing with things like syntax errors (due to the absence of any built-in integrity checks); but that also isn’t a claim that Microsoft’s implementation of a registry actually is more efficient.
Brendan,
Good because I don’t think we can make such claims without measuring them. BTW I actually like structured data, but even highly tuned databases can prefer doing what’s called a full table scan over doing several hundred calls to an index and I think the same probably applies here. While I’m sure we can improve on text files, I suspect the registry is still going to be slower because of all the syscalls.
I agree with you on the lack of a standardized integrity checks, but that also applies to the registry. The registry has so many foreign keys that have no integrity checks…it can be filled with garbage and we’re none the wiser.
Alfman,
“Foreign key” enforcement on registry would probably be a disaster. Because they can refer to items on file systems, removable media, or even the network, it would need to be an overly complex and error-prone system.
Even when they are internal references, it might just be cheaper to have stale data around.
Given they implemented transactions logs and several levels of backups, I would say the integrity of the storage itself is no longer a concern.
(Again, not the best design, but it works okay).
sukru,
Well, I think that’s only true if your retrofitting it into a legacy OS. Nothing would be consistent or compatible. Everyone would turn it off to be compatible with legacy software and it would end up being brushed to the side as a failure. But if it were designed properly from the ground up including the ecosystem it would actually solve a lot of the problems we have now.
Actually I have been calling for such features to be built into operating systems for decades. If we had robust integrity including foreign keys at the OS level you could completely uninstall an application and be confident that there’s no lingering junk. The need to reinstall to obtain a clean system could be eliminated for good. Many of these mechanisms have evolved into the massive enterprise databases we use daily and they work. We could put them to great use in modern operating system design as well.
I concede that I don’t honestly expect any dominant operating systems to change at this point. For better or worse we usually end up having to live with what we’ve got rather than what we could have, not because we cannot do better, but because of the enormous scope of replacing everything. The hurdles are too great to gain acceptance.
I disagree. Rolling the entire system back is usually a last resort for me. More often I just want bits and peices to be restored.
Alfman,
I get what you say, and wish that was possible. But given there is no standard package system in Windows, nor a way to force applications to declare configurations (which does not even exist in general purpose Linux btw), this would be an intractable task.
Say, you installed a Photoshop plugin (making this up), and then later installed a new Photoshop instance, and had the prior plugin to register itself. Let’s say Photoshop has its own plugin “registry”. How can Windows roll back the initial install of that plugin, without breaking the newer Photoshop?
Anyway, they have a “general rollback” of the OS, including NTFS transactions, which is objectively better than what we had to go through before.
sukru,
I want to make a distinction between the difficulty of implementing the feature itself, which is feasible, versus the difficulty of actually getting the entire windows ecosystem to adopt it. I’ll grant you that at this point It certainly seems impossible to get the entire windows software catalog on board. My point was that if it had been incorporated & standardized into windows from the get go it wouldn’t have been a big deal and we would all have it now. In other words the barrier is not developing integrity enforcement, but rather getting the vast pre-existing software ecosystem to use it.
I think Microsoft’s MSI installer technology has some of the needed foundations, but obviously it’s missing OS integrity checks and enforcement. Linux packages could get us close too given their ubiquitous on most linux distros, but there too linux lacks OS level support to enforce integrity constraints like a database does.
I don’t see an issue, dependency scenarios like this and more complicated ones are regularly encountered by package managers. But I think we would gain more stability and features at the OS level.
We’ve been talking about features in fairly basic terms, but I actually envision this evolutionary path opening up much more impressive git-like capabilities with a very powerful feature set for tracking/merge/shelve/etc individual changes that work in conjunction with the operating system’s integrity enforcement. Having OS enforcement means the state of packages can actually be robustly trusted unlike today. It could change the face of tech support and having robust tools to identify all the local changes and even having the ability to “blame” specific changes that are causing a fault.
A non-selective general rollback is implicitly destructive though, loosing all the “good” changes since the backup. It would be like having to roll back an entire code base from backup (extremely negative side effects) when what you really want is to undo a specific change while keeping all the rest of your work intact. In principal there’s no reason operating systems need to be this clunky, but in windows it’s mostly a byproduct of how difficult and painful it is to do anything with the registry. Even when you restore registry, you risk it getting out of sync with the state of applications.
I don’t want to be overly dramatic about it, but these gripes are serious pain points. When people bring me their systems to clean up/fix/migrate/etc, I would like to be able to back up just desired applications and move them to a new/clean system, often with a new OS. The registry makes this a royal pain. It’s easier NOT using registry backups and just going down the reinstall everything path instead. IMHO this is clear evidence that the tooling just isn’t very good.
Edit: Sorry about the long winded post. I hope it doesn’t come across as confrontational. I just think this is a very interesting discussion!
Alfman,
No worries. Our discussions are nice even though we would disagree from time to time. And, I tend to be stubborn anyway.
In the registry you have a path, a key and a value, right?
in a file system you have a path, a file name and file contents, right?
so instead of the registry key of
Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Notepad\fMatchCase
containing value 0x00
you would have a file like
C:\Windows\Config\Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Notepad\fMatchCase
that contained 0x00.
Make sense?
Bill Shooter of Bul,
Indeed, the hierarchies are functionally the same. Heck, there’s even a linux file system for the windows registry!
https://github.com/jbruchon/winregfs
“Why cant microsoft just remove it now in the age of ssd’s and processors faster than a 486?”
Because it’s vastly superior to config files. You don’t have much real Linux experience if you haven’t seen your config files clobbered into your machine being in an unbootable state.
dark2
That’s debatable, but more importantly it’s beside the point. When your house stinks, should you really be justifying it by saying someone else’s house stinks more? Even if you were right, it’s hardly something to boast about.
How is it besides the point? You’re advocating for an entire SQL database to store settings, while the status quo is either the Windows flat database (which while you might find annoying is rather fast, efficient, and not prone to breaking due to it’s atomic key/value pair nature; [far less than 1% of users for the last 10 years have ever had to manually edit the registry]), or config files (which as stated, are often corrupted even through automated tools). Interestingly an already linked article has explained why going full sql database would just make the problems of the registry even worse with developers storing all their data in it, and attempting to enforce referential integrity on uninstalled programs/missing files.
dark2,
Because it’s a straw man/whataboutist argument. The OP offered legitimate criticisms for windows and you want to counter with attacks on linux. Regardless of your opinions about linux, that’s not a rebuttal for user gripes against windows.
In 25 + years, I’ve never had that happen. Maybe just luck? I guess I’ve had odd issues with fstab, or xwindows, that I kind of intentionally caused because I knew I could just boot via floppy/cd/usb drive and fix once I had the new device I was putting in. I think you can do the same with windows registries, but thats not as straight forward. I might have done that once by mounting the drive in a separate computer and modifying the registry that way. But its a bigger pain, imho.
The nice thing about most Mac OS or some linux applications ( basically zero standards as I’m sure we’re all aware) is the ability to just copy an application’s directory to a separate backup, and boom you’ve got the app and all its data backed up. Doing that on windows apps that use the registry is very very difficult, even if you are the developer of that app!
Bill Shooter of Bul,
+1 This!
Even on windows it used to be trivial to backup/migrate applications. Microsoft’s own software including office worked this way. Most software did not adopt the registry right away, but gradually as they did I ended up having to ditch this simple backup strategy because it just didn’t work anymore.
While it is “nifty”, it is also a very common practice.
Appending files has been done in nvidia shell scripts with their Linux driver binaries, RAR self extractors with archive data, EXE files with almost anything (self contained games, overlays, resources). And resource embeds are actually standard part of Windows API.
The trick in this new article is Powershell finally being actually at level of UNIX command line versatility. On the more “civilized” side, “head” and “tail” commands would do the same thing.
Appending works, I wonder if they could have used an alternative data stream in NTFS to achieve the same thing. I don’t understand whey they couldn’t have, except maybe that would affect portability.
Bill Shooter of Bul,
NTFS streams could work, but as this is a proof of concept how can an attacker create ntfs streams via an email payload? Is there an archive format that creates them and then how would they get executed? Obviously a program can set this all up but when you run the attacker’s program they’ve already won. This other stuff seems superfluous.
Yes, there are a number of archive formats like 7zip that will include them. You’d have to launch it the same way this one does with some powershell commands. I just think, maybe incorrectly, that alternative data streams are usually missed by most people.
Bill Shooter of Bul,
I didn’t know that.
It looks like rar supports it, but you need to explicitly enable it on the command and in the GUI.
https://superuser.com/questions/156971/ntfs-alternate-stream-in-compressed-archive
I’m finding lots of feature requests on 7zip’s forums…but I had trouble finding definitive answers for the 7z format. It looks like it might work through the other formats z7ip supports.
https://sourceforge.net/p/sevenzip/discussion/45798/thread/4da21400/
https://sourceforge.net/p/sevenzip/discussion/45797/thread/eefcd83f/
I agree, but I thought the raison d’être for the article is that the code is executed automatically without the user having to type in or run other commands.
Thanks! (NOT!) Now if the aholes didn’t know about this they do now and so there will even more variety of viruses out there. They probably were already but now people that might have been as smart now see another way to screw with people and do bad things to them. Thanks a lot! NOT!
You know those capital letters aren’t free, right?
Eh, responsible disclosure. I think the reality is antivirus would catch on quickly and find the malware signature in the registry entry. Its just kind of a neat vector most haven’t thought of. I’ve abused this property in other file formats, appending extra data I needed but for various reasons couldn’t introduce a new file.