The answer can often be found in a fairly simple way: Do a call stack review!
Let's walk through an example showing this type of investigation with my favorite 32-bit user-mode debugger: OllyDbg
We will replace the first assembly command of the shellcode with INT 3 (a break-point command). This will cause the debugger to halt the execution before the shellcode gets a chance to execute. So, we will be able to review the memory state of the process at that point.
Here we see the shellcode before we replace the first assembly command with INT 3 opcode 0xCC:
And here we see what it looks like when we add INT 3 to the shellcode:
Below, we can see the state of memory after we execute the shellcode. Note that we changed the memory address to a long-address format, because this enables us to see who redirected our INT 3 breakpoint:
And we can see that at address 0x664C888 we have a reference to 0x76B21106. When we disasm the location, we can see that it was the indirect CALL DWORD PTR[EAX+4] inside the ATL.DLL module that redirected us to our INT 3.
A side note: Sometimes a JMP assembly command can be used to redirect execution to the shellcode. In that case, doing a call stack review will not help you much. In those instances, the Run trace feature of OllyDbg can and will help you!
Security Researcher: Moti Joseph