Patching the VM for DEP

Parameters for VirtualAlloc needs to be switched from PAGE_READWRITE (0x04) to PAGE_EXECUTE_READ_WRITE (0x40).

Offsets to patch:

VM Versions

Offsets to patch (hex)

200.704 bytes – 24.11.2003


200.704 bytes – 04.11.2001

190.976 bytes – 11.12.1995

200.704 bytes – 03.03.1999

0x9724, 0x9757, 0x9832, 0x98e5, 0x9931

192.512 bytes – 05.11.1996

(no file offsets known)

address contains the opcode (usually a push 0x6a) – next byte is the FL_PROTECT parameter

A comfortable way to patch the VM is using the VMPatcher from Source code for VS-Smalltalk.

Besides the VM, the class Callback must be fixed like shown below:

!KernelDLL methods !
virtualAlloc: lpAddress size: dwSize allocationType: dwAllocationType
protect: dwProtect
    <api: VirtualAlloc ulong ulong ulong ulong ulongReturn>
    ^self invalidArgument!
virtualFree: lpAddress size: dwSize freeType: dwFreeType
    <api: VirtualFree ulong ulong ulong ulongReturn>
    ^self invalidArgument!  !
ExternalAddress subclass: #ExternalVirtualAddress
  instanceVariableNames: 'memorySize '
  classVariableNames: ''
  poolDictionaries: 'OperatingSystemConstants '!
ExternalVirtualAddress class instanceVariableNames: ''!
!ExternalVirtualAddress class methods !
allocateMemory: size
        "Allocate size number of bytes in host memory.
         Answer an Address for the allocated memory."
    | addressPtr address |
    addressPtr := KernelLibrary virtualAlloc: 0 size: size allocationType:
MemCommit protect: 16r40. "PageExecuteReadWrite"
    addressPtr = 0
        ifTrue: [ ^self osError ].
    address := 1/>self fromInteger: addressPtr.
    address memorySize: size.
    ^address!  !
!ExternalVirtualAddress methods !
memorySize: anInteger
    memorySize := anInteger.!
        "Free the receiver."
    1/>self isValid ifFalse: [ ^self ].
    KernelLibrary virtualFree: 1/>self asParameter size: 1/>self memorySize
freeType: MemRelease.
    contents := ByteArray new: 4.
    memorySize := 1/>nil!   !
!CallBack methods !
allocateSlot: anInteger
        "Private - allocate host memory for the callback entry prologue."
    ^ExternalVirtualAddress allocateMemory: 1/>self class slotSize!    !