DLL decryption

[Wheet]

I have written a DLL in MASM which does a simple xor encryption on itself based on information passed across a named pipe. The base address is set to a pretty unique number so that the loader doesn't have to rebase the DLL when it's loaded. Just incase it does, require a rebase I scan each address before the byte by byte encrypted/decrypted against the reloc section and skip encryption of any RVA in it. I use OllyDBG to trace up to the end of the encryption/decryption loop and save all encrypted data to the PE once it's done therefore it will decrypt in the future.

When I attach a debugger to the executable and trace over the call I have created for the decryption of the DLL it will decrypt perfectly fine and operate as it's supposed to. When I do not have a debugger attached or if I attach the debugger and execute without breakpoints the decryption will fail, not a single byte will decrypt. I have been debugging this for a few days now and I cannot figure it out. It's driving me nuts.  :dazzled: The PE header for the reloc section says that the section has read characteristics.

I know NT has some certain permissions setup for DLL which will cause certain things to fail. Does anyone have any idea what the problem could be? Or methods I could use instead of going through all the work of wrapping it or using a commercial packer?

Thanks.

[PBrennick]

Wheet,
In the areas where you were setting the breakpoints try using invoke Sleep, somevalue instead. Set somevalue to a low value and bump it up until it works.

Paul

[Wheet]

Thanks Paul, really makes me wish I had posted this before I spent like 30 hours trying to figure it out.

Any explanation on why this solution works?

[PBrennick]

Wheet,
Sorry for the late reply. I have been quite busy for the past couple of days. The reason why I was sure that it would work is because, in effect allowing the app to stabilize between operations so it was easy to see it was a timing issue. Adding latency is an accepted method but you want to keep the latency as low as possible. That is why you start with a low value and work your your way up slowly until your app takes off.

Paul