October 29, 2006

Fun with NTVDM

The other day I discovered by accident that you can actually execute plain text files (or any file with arbitrary binary content for that matter) in Windows or, more exactly, in the 16-bit MS-DOS subsystem provided by the system file NTVDM.exe in 32bit Windows systems. I assumed Windows would check the magic number (first bytes) of a file before executing it, to make sure it is an executable before actually executing it but it seems this is not the case: it will happily interpret as binary code any file, be it a real executable or not.

Not that this piece of knowledge is particularly useful (you don't want to execute ASCII files anyway) but I had some fun with it and it may save you some time if you can recognize the error messages. The bottom line is: if you're trying to execute what you think is an executable and you're getting weird error messages, consider the possibility that what you are executing is not an executable after all.

Anyway, this is the story. I had been given a .EXE file that was supposed to be a malicious program for Windows. I copied it over to a lab Windows machine and tried to execute it, first by double-clicking its icon in Windows Explorer and then by invoking it from a command prompt window (cmd.exe), with identical result: an error message like the one shown in the following picture.


NTVDM (NT Virtual DOS Machine) is a process that "simulates a 16-bit Windows environment complete with all of the DLLs called by 16-bit Windows applications". That's how 16 bit Windows/DOS applications are run in modern (32/64 bit) Windows releases.

OK, but why the illegal instruction? It turns out the file was the wrong file. Actually, it was a copy of a simple ASCII text file, renamed to .EXE by mistake, and yet the system had tried to interpret it as a real EXE file. No wonder it found an illegal instruction! However, depending on the contents of the ASCII file the result might have been different.

Then, just for fun, I tried the following. I created a few simple ASCII files using notepad.exe, each containing just one word (not even a carriage return) and I named each file by the word it contained and I added the extension ".exe" to all of them. Like this:
---
C:\test>type Hello.exe
Hello
C:\test>type David.exe
David
C:\test>type Raul.exe
Raul
C:\test>type Jorge.exe
Jorge
C:\test>type Problem.exe
Problem
C:\test>
---

Then I executed them, with a variety of results:
---
C:\test>Hello.exe
[NOTE: Pop-up error message like the one shown above (illegal instruction)]
C:\test>David.exe

C:\test>Raul.exe

[NOTE: For David.exe and Raul.exe, no output, just a carriage return and the prompt again]
C:\test>Jorge.exe

Divide overflow

C:\test>Problem.exe
[NOTE: Here the command prompt window freezes]
---

David.

3 Comments:

Anonymous Anonymous said...

Actually NTVDM does check for the magic sequence (and by this you probably mean MZ) and if it finds it, loads it as an exe file (which means it maps the file accordingly in memory). But if it doesn't find the MZ marker (and the file is smaller than 64k - I think) it assumes that it's a good old com file and tries to execute it that way. That's why your text file gets executed. Try adding MZ at the begininning and you'll see that nothing happens because it will try to parse and and will find an illegal header.

Actually windows doesn't care all that much about the file extension. As long as it's one of the following: exe, com, pif (and maybe lnk - I'm not sure), it will try to determine heuristically how the file should be executed. So if you rename notepad.exe to notepad.com it will still work because it will detect upon execution that it's a Win32 PE executable and run it accordingly.

Best regards,
Cd-MaN

7:00 AM  
Blogger Raul Siles said...

Who said Windows is not a smart OS! Analyzing the facts:
- If you execute David or Raul it says nothing. There is nothing to say about the two of us; we're just fantastic! :-)
- If you try to execute Jorge, it overflows. I understand Windows, sometimes it is difficult to manage Jorge's rebelliousness ;-) ;-) ;-)
- If you try to execute a problem (BTW, why on earth did you try to execute a problem in Windows? :-D), then it hangs, because Windows doesn't like to solve problems :-)

A bit of humor here! ;-)
Raul

9:22 PM  
Blogger Yuhong Bao said...

Raul Siles: It is not Windows doing this, it is the x86 processor executing ASCII text as 16-bit code. Disassemble all of these using DOS DEBUG shipped with Windows.
Here is an article from slashdot that talks about this:
http://it.slashdot.org/story/09/11/23/1837238/English-Shell-Code-Could-Make-Security-Harder

7:28 PM  

Post a Comment

<< Home