Using MASM on Mac or Linux via Wine

This semester, I am taking a class on assembly programming using Kip Irvine’s book “Assembly Language for Intel-Based Computers.” Unfortunately, the book should really be called “Assembly Language for Intel-Based Windows Computers,” as it is written for Microsoft Macro Assembler (MASM). It mentions that the programs in the book could be converted to TASM assembly fairly easily, but they will not run out of the box.

I could run MASM on a Windows virtual machine, but that would be fairly heavy to have running whenever I want to work on assembly. Instead, I wanted to see if I could run MASM with little overhead using Wine. It turns out that this is fairly easy to do!

Install Wine

Good instructions for installing Wine on Mac are available here. Instructions for Ubuntu are available here. The install process for other distros should be fairly straightforward as well.

The TL;DR for intsalling on Mac is to install XQuartz and wine via homebrew:

brew install Caskroom/cask/xquartz wine

Create a new wineprefix

Commands executed using wine are executed in “Wine prefixes,” which are virtual Windows environments of sorts. By default, the ~/.wine prefix is used; configuration for this environment is stored in ~/.wine/*.reg files, and the C:\ file tree is stored in ~/.wine/drive_c.

We could run MASM in the default wine prefix and it would work perfectly fine. However, we will be setting Windows environment variables that could potentially interfere with other programs running on Wine. If you plan on using Wine for anything else, it is best to install MASM in its own Wine prefix. You can create a new prefix as follows:

WINEARCH=win32 WINEPREFIX=~/wine-masm winecfg

Wine will initialize a new Windows file system tree at ~/wine-masm/drive_c and open a window for you to configure the system. The defaults are fine, so you can close the Wine Configuration window that appears.

Download and extract MASM32

I downloaded MASM from http://www.masm32.com/. You can download and run the installer as follows:

# You may want to install wget and unzip through your package manager if
# you don't have them
wget http://website.assemblercode.com/masm32/masm32v11r.zip
unzip masm32v11r.zip
WINEPREFIX=~/wine-masm wine install.exe

The first steps of the installer are fairly self-explanatory:

After clicking OK, the installer starts assembling/linking some libraries and outputs its progress in the terminal:

At one point, the installer asks if I want to overwrite msvcrt.exp. I’ve tried it with both yes and no, and I don’t think it matters.

A dialog appears asking if you want to create a shortcut to the MASM editor:

We aren’t using the Windows desktop, and we probably won’t even be using the MASM editor (I much prefer Sublime or vim), so click No.

At the end, the MASM editor appears. You can close this, or check it out if you’re interested. You can always open it in the future by running WINEPREFIX=~/wine-masm wine 'C:\masm32\qeditor.exe'.

At this point, the installation is complete, and you should be able to run the MASM assembler:

$ WINEPREFIX=~/wine-masm wine 'C:\masm32\bin\ml.exe'
Microsoft (R) Macro Assembler Version 6.14.8444
Copyright (C) Microsoft Corp 1981-1997.  All rights reserved.

usage: ML [ options ] filelist [ /link linkoptions]
Run "ML /help" or "ML /?" for more info

Download and extract Irvine’s files (optional)

If you are using Irvine’s textbook, you will want to download his example and include files. The files are available on his website at http://www.kipirvine.com/asm/examples/index.htm. I am using the 5th edition textbook, so I downloaded the files from http://www.kipirvine.com/asm/examples/IrvineExamplesVS2008.zip:

wget http://www.kipirvine.com/asm/examples/IrvineExamplesVS2008.zip
unzip IrvineExamplesVS2008.zip -d IrvineExamplesVS2008
cp -r IrvineExamplesVS2008/ ~/wine-masm/drive_c/Irvine

Irvine provides some scripts that (quite irritatingly/inflexibly) expect this download to be extracted to C:\Irvine:

cp IrvineExamplesVS2008/ ~/wine-masm/drive_c/Irvine

If you are using a newer edition of the book, the example files are packaged in a .msi installer. You can download/extract as follows:

wget http://www.kipirvine.com/asm/examples/Irvine_7th_Edition.msi
WINEPREFIX=~/wine-masm wine msiexec /i Irvine_7th_Edition.msi

Accept all the default options.

Set environment variables

We need to set the Windows PATH, INCLUDE, and LIB environment variables so that when we are assmebling/linking, we can find the MASM binaries/includes/shared libraries more easily. To do this, open regedit:

WINEPREFIX=~/wine-masm wine regedit

Browse to HKEY_CURRENT_USER/Environment. Add a new string value (right click the right-hand pane, New > String Value) named PATH. Double click this new value and enter the following:

%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\masm32\bin

Also create an INCLUDE key with value C:\masm32\include;C:\Irvine and a LIB key with value C:\masm32\lib;C:\Irvine. (If you are just trying to run masm on *nix and aren’t using Irvine’s book, then you can omit the C:\Irvine parts of those keys.)

At this point, you can call masm without needing to specify full binary or include paths:

$ export WINEPREFIX=~/wine-masm
$ wine ml -nologo -c -coff -Zi AddSub.asm
 Assembling: AddSub.asm
$ wine link /NOLOGO /SUBSYSTEM:CONSOLE /ENTRY:main \
    /LIBPATH:'C:\Irvine' irvine32.lib kernel32.lib user32.lib AddSub.obj
$ wine AddSub.exe

  EAX=00030000  EBX=00401000  ECX=0032FF03  EDX=000000D0
  ESI=7B84A20B  EDI=7FFDF000  EBP=0032FEE8  ESP=0032FED4
  EIP=00401014  EFL=00000206  CF=0  SF=0  ZF=0  OF=0  AF=0  PF=1

Write a bash function to easily assemble/link/run .asm files

At this point, we could be done. However, specifying all the masm flags can become a pain, and we could easily write a bash function to take care of this. In my ~/.bash_profile, I have the following:

export WINE_MASM_DIR=~/wine-masm
function masm() (
    # Stop on errors
    set -e

    # Use the correct wine directory
    export WINEPREFIX=$WINE_MASM_DIR

    # Get the path to the file without a .asm extension
    FILENAME="$(basename "$1")"
    EXTENSION="${FILENAME##*.}"
    shopt -s nocasematch   # String case-insensitive comparison
    if [[ "$EXTENSION" = "asm" ]]; then
        FILENAME="${FILENAME%.*}"   # Remove extension
    fi
    UNIX_PATH="$(dirname "$1")/$FILENAME"

    # Convert forwards slashes into backslashes
    WINDOWS_PATH=$(echo "$UNIX_PATH" | tr '/' '\')

    # Assemble file
    wine ml -nologo -c -coff -Zi "$WINDOWS_PATH.asm"

    # Link files
    # Notes: Irvine's asm32.bat script includes a /DEBUG flag. I found that
    # including this flag causes a link fail:
    #   LINK : fatal error LNK1000: unknown error; consult documentation for
    #   technical support options
    # Therefore, I have omitted it. Additionally, the default entry point
    # for Wine or perhaps later versions of Windows appears to be
    # mainCRTStartup instead of main (see http://stackoverflow.com/a/12391264),
    # so we need to manually specify main as the entry point with /ENTRY.
    wine link /NOLOGO /SUBSYSTEM:CONSOLE /ENTRY:main \
        /LIBPATH:'C:\Irvine' \
        irvine32.lib kernel32.lib user32.lib "$WINDOWS_PATH.obj"

    # Run the linked executable
    wine "$WINDOWS_PATH.exe"
)

Feel free to tweak this function as necessary.

Once you “reload” your bash profile (. ~/.bash_profile), you can assemble/link/run .asm files in one command:

$ masm AddSub.asm
 Assembling: .\AddSub.asm

  EAX=00030000  EBX=00401000  ECX=0033FF03  EDX=000000D0
  ESI=7B84A20B  EDI=7FFDF000  EBP=0033FEE8  ESP=0033FED4
  EIP=00401014  EFL=00000206  CF=0  SF=0  ZF=0  OF=0  AF=0  PF=1

Conclusion

Wine is really handy when it works – and it works quite well in this case, without any DLL overrides or fancy configuration or what not. We are able to run MASM and assembled binaries with little overhead, and we can use our host shells and editors without needing any fancy tricks!