There is no such thing as "DOS mode" in Mac OS X. Most likely you're trying to run a Windows application, which is not compatible with Mac OS X. Try and find a Mac version of the software. If there isn't one, look into solutions for running Windows applications on a Mac, such as:
* Virtual machine software, such as VirtualBox or VMWare Fusion. This is the best solution in most cases. Put simply, it lets you run Windows inside of Mac OS X, creating a separate virtual computer that you can use to run Windows software. The downside is that programs won't benefit from all the computer's memory and processing power. It also requires a copy of Windows, which you'd need to obtain from somewhere.
* Boot Camp. This is an official utility made by Apple that lets you run Windows on your Mac. Unlike a virtual machine, Windows runs by itself just like if you had a Windows machine, so programs have the full power of the system to work with. The downside here is that it basically requires you to split your hard drive into two separate drives (no, not physically cut the drive; it's just a software thing) so you'll have less disk space available to both. Also, you can only run one of the two operating systems (Windows or Mac OS X) at once, so you can't run applications for both at the same time, and in order to switch between the two you need to reboot. Like before, you need a copy of Windows for this.
* Wine. (Not the beverage, it's a program you can download.) In general, I wouldn't recommend this. It's not dangerous or anything, but a lot of programs won't work on it, or if they do, they'll have a bunch of bugs. I've rarely seen a program work perfectly on Wine. But in the semi-rare cases when it does work, it's a clear winner. Running a program with Wine lets it run directly in Mac OS X with the system's full processing power. And you don't even need a copy of Windows. But again, for most programs it won't work perfectly, if at all.
The reason it mentions DOS mode is because, since Mac OS X doesn't recognize the "EXE" extension, it opens the file in TextEdit by default. However, EXE files are not designed to be opened in text editors, so it shows up as a bunch of garbage text. The "DOS mode" text happens to appear at the beginning of most EXE files, to be displayed in case the user tries to run it in MS-DOS. But in your case, it's not being displayed as an error message; it's just being displayed because it's part of the file you opened.
Knowing Apple's tendency to make things simple for novice users, I'm surprised they left this confusing behavior in, given that opening an EXE file on a Mac is something that's likely to happen if the user doesn't know it's incompatible. They should really make it display a more accurate message, like "This application requires Microsoft Windows", and give information like I gave before about running Windows applications on a Mac.