It sounds like the file is an encrypted self extracting file. In other words it needs a password to confirm decryption, decompression and installation.
From Wikipedia:
A file with the extension .dmg uses a disk image format commonly found on Mac OS X. The format allows secure password protection as well as file compression and hence serves both security and file distribution functions. Its most common function is the distribution of software over the Internet. When opened, DMG files are "mounted" as a drive within the Finder.
DMG files can be easily created (with or without encryption) using utilities that are included in OS X: Disk Utility in Mac OS X v10.4 and 10.3 or Disk Copy in earlier versions. These utilities also use DMG files as images for burning CDs and DVDs. DMG files may also be managed via the command line using the hdiutil utility.
DMG files are published with a MIME type of application/octet-stream. As many web publishers tend to be less knowledgeable about Apple-specific file types, this MIME type is not set, often resulting in the user experience of attempting to download DMG files as text directly to the browser window, forcing the use of an option-click or a similar workaround to download the file. For this reason, DMG files are often distributed as bzip2 (.dmg.bz2) or ZIP (.dmg.zip) files. These wrappers typically don't compress the file further, but rather help ensure that the files are handled correctly by the server and browser software.
Utilities exist to convert DMG files into ISO images, however, DMG files do often contain other types of disk images such as HFS+ volumes.
Just my .01
-dh