Strange Behavior When Your .NET Application Uses ActiveX/COM and Data Execution Prevention (DEP) is Enabled?

This issue was extremely obscure, but I wanted to document it anyway. The moral of the story is, if your .NET application is using an ActiveX/COM object, and is behaving strangely on Vista with DEP enabled, try to get DEP turned off and see if anything changes. In our case, a third party ActiveX/COM object was trying to pop up a window, and DEP was killing it, causing strange behavior in the application we were debugging.

Details...

We've got a MagTek MICRImage check reader, and we've written a program in .NET that communicates with it, and everything works fine. We are using SaxComm8.ocx to communicate with the reader (an ActiveX object that is using AxInterop for .NET use).

Next thing you know, we are trying to debug the application, and the checkreader no longer triggers any "check read" events when you scan a check. Communication otherwise works fine with the reader. We'd checked everything to do with your serial ports.

In our case, it turns out it was a combination of the SaxComm8.ocx code being overwritten by a previous version we had that was a trial only. Typically this trial version pops up a warning box about the trial period. But in Vista with DEP running, this dialog was never appearing. Most likely this was then causing the SaxComm8 object to go into a weird state, no longer communicating with the reader.

Finally found that DEP was causing the problem. Then I discovered Vista would not allow me to disable DEP for this app (WHY?!). Used the command bcdedit /set nx AlwaysOff, and restarted. Of course, now DEP is totally disabled, but finally I was able to see the trial notice popup and realize what was happening.

Once I overwrote all trial copies of the .ocx file, I did a regsvr32 /u saxcomm8.ocx to unregister it, and a regsvr32 saxcomm8.ocx, to register it, and now everything works!