When does a Virtual Machine stack overflow happen?
Usually you will get a stack overflow if you write recursive code that does not terminate correctly.
If the VM detects a stack overflow, an internal flag is being set.
The stack overflow triggers a walkback, which calls Process class>>#queueWalkback:makeUserIF:resumable:.
#queueWalkback:makeUserIF:resumable: will call Process>>#dropSenderChain during the error processing.
#dropSenderChain will clear the stack and will set up a new stack (process). It also clears the internal flag, that has been set during a stack overflow.
If an error occurs before the #dropSenderChain has been done, additional stack frames must be created and additional stack overflow checks will occur. If the internal “stack overflow” flag is still set, a virtual machine stack overflow will appear.
One possible reason that will trigger a virtual machine stack overflow are the senders of #osError in CursorManager. If your app does a lenghty computation and the Windows screensaver will be activated, many cursor related API calls will cause an OS error.
During the error handling in #queueWalkback:makeUserIF:resumable: “CursorManager normal change” will cause further walkbacks causing the stack to fill until the stack overflow will be triggered.