Patching the VM for DEP

Parameters for VirtualAlloc needs to be switched from PAGE_READWRITE (0x04) to PAGE_EXECUTE_READ_WRITE (0x40).

Offsets to patch:

VM Versions

Offsets to patch (hex)

200.704 bytes – 24.11.2003


200.704 bytes – 04.11.2001

190.976 bytes – 11.12.1995

200.704 bytes – 03.03.1999

0x9724, 0x9757, 0x9832, 0x98e5, 0x9931

192.512 bytes – 05.11.1996

(no file offsets known)

address contains the opcode (usually a push 0x6a) – next byte is the FL_PROTECT parameter

A comfortable way to patch the VM is using the VMPatcher from Source code for VS-Smalltalk.

Besides the VM, the class Callback must be fixed like shown below:

[codesyntax lang=”smalltalk”]

!KernelDLL methods !

virtualAlloc: lpAddress size: dwSize allocationType: dwAllocationType
protect: dwProtect
    <api: VirtualAlloc ulong ulong ulong ulong ulongReturn>
    ^self invalidArgument!

virtualFree: lpAddress size: dwSize freeType: dwFreeType
    <api: VirtualFree ulong ulong ulong ulongReturn>
    ^self invalidArgument!  !

ExternalAddress subclass: #ExternalVirtualAddress
  instanceVariableNames: 'memorySize '
  classVariableNames: ''
  poolDictionaries: 'OperatingSystemConstants '!

ExternalVirtualAddress class instanceVariableNames: ''!

!ExternalVirtualAddress class methods !

allocateMemory: size
        "Allocate size number of bytes in host memory.
         Answer an Address for the allocated memory."

    | addressPtr address |
    addressPtr := KernelLibrary virtualAlloc: 0 size: size allocationType:
MemCommit protect: 16r40. "PageExecuteReadWrite"
    addressPtr = 0
        ifTrue: [ ^self osError ].
    address := self fromInteger: addressPtr.
    address memorySize: size.
    ^address!  !

!ExternalVirtualAddress methods !

memorySize: anInteger

    memorySize := anInteger.!





        "Free the receiver."
    self isValid ifFalse: [ ^self ].
    KernelLibrary virtualFree: self asParameter size: self memorySize
freeType: MemRelease.
    contents := ByteArray new: 4.
    memorySize := nil!   !

!CallBack methods !

allocateSlot: anInteger
        "Private - allocate host memory for the callback entry prologue."
    ^ExternalVirtualAddress allocateMemory: self class slotSize!    !