The flaw exists in the Virtual DOS Machine, which is a system that allows Windows NT to run DOS and 16bit applications on 386 (and up) machines. Google security team member Tavis Ormandy discovered a vulnerability in the VDM and reported it to Microsoft in the summer of 2009, but since it still hasn't been patched, he went full-disclosure.
Basically, what happens is that a 16 bit program can escalate its privileges and gain unrestricted access to the entire system, but as far as I can tell (please correct me if I'm wrong) this is a local exploit: you need physical access to the system. This usually means normal users have relatively little to worry about, but of course, it can be used in social engineering attacks, or someone might find a way to exploit it remotely.
The vulnerability is present in all 32bit versions of Windows NT, starting with NT 3.51 and ending with Windows 7. Since 16bit support was dropped from the 64bit versions of Windows, users of Windows 64bit are not affected. The work-around is extremely simple and straightforward: disable the 16bit subsystem on 32bit machines.
This can be done one of three ways (but they all do the same thing: edit the registry). First, you can use the Group Policy Editor to enable the "Prevent access to 16-bit applications" in
Computer Configuration\Administrative Templates\Windows Components\Application Compatibility. Second, you can also simply go rogue and edit the registry directly (backup! backup!) by placing a key in
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\AppCompat with a D-Word value of
VDMDissallowed = 1.
The last method automates it all: create a text file called
vdmdisallow.reg, and paste the following into the file, and double-click it:
Again, 64bit users needn't worry. All this does raise a question we've sure raised before: at what point should Microsoft drop support for applications? How many people still use 16 bit applications? Wouldn't it have made more sense to have it disabled by default on 32bit systems, and then enable it as soon as Windows detects someone is trying to load a 16bit application?