Calling VS 2005 DLL

[Astro]

Hi,

I'm trying to call a VS 2005 DLL but I get a memory access violation every time.

Talking to someone more knowledgable that I, they said I need to manually link to the function, but they didn't know how to do this in assembler.

The DLL is from Crypto++, FIPS-140 build, V5.3.0. I'm trying to call the function "CalculateDigest".

Another option is to use another library, but this has the advantage of already being FIPS certified.

http://www.cryptopp.com/

Code Select Expand

.386
.model flat,stdcall

include \masm32\include\windows.inc
include \masm32\include\kernel32.inc
include \masm32\include\user32.inc
includelib \masm32\lib\kernel32.lib
includelib \masm32\lib\user32.lib

.data?
hCryptoLib dword ?
hFunc dword ?

;---------------------------------------------
; SHA-256
;
Digest byte 32 dup (?)

.code
CryptoLib db "cryptopp.dll",0
;---------------------------------------------
; Functions
;
FIPSEnabled db "FIPS_140_2_ComplianceEnabled",0
ordFunc2330 dword 2330 ; FIPS_140_2_ComplianceEnabled
ordFunc1899 dword 1899 ; AlgorithmName - SHA256
ordFunc2021 dword 2021 ; CalculateDigest - SHA256

;---------------------------------------------
; SHA-256
;
HashMe byte "Data to hash",0

;---------------------------------------------
; Messages
;
app byte "Test",0
LIBFail byte "LoadLibrary Failed",0
FIPSFail byte "FIPS Proc Failed",0
FIPSFail2 byte "FIPS Call Failed",0
Pass byte "Working!",0

start:

push offset CryptoLib
call LoadLibrary
mov hCryptoLib,eax

test eax,eax
jnz @F

push 0
push offset app
push offset LIBFail
push 0
call MessageBox

ret
;===========================================================

@@:
push ordFunc2330
push hCryptoLib
call GetProcAddress
mov hFunc,eax

cmp eax,0
jz FAIL

call hFunc

cmp eax,0
jz FAIL2

push ordFunc2021
push hCryptoLib
call GetProcAddress
mov hFunc,eax

push 13
push offset HashMe
push offset Digest
call hFunc ; virtual void CalculateDigest (byte *digest, const byte *input, size_t length)
; 2021  7E4 000283F0 ?CalculateDigest@HashTransformation@CryptoPP@@UAEXPAEPBEI@Z

push 0
push offset app
push offset Digest
push 0
call MessageBox

push hCryptoLib
call FreeLibrary

ret
;===========================================================

FAIL:
push 0
push offset app
push offset FIPSFail
push 0
call MessageBox

push hCryptoLib
call FreeLibrary

ret
;===========================================================

FAIL2:
push 0
push offset app
push offset FIPSFail2
push 0
call MessageBox

push hCryptoLib
call FreeLibrary

ret
;===========================================================

end start

Best regards,
Robin.

[BlackVortex]

tl;dr

Too boring to follow all the conditional jumps. What's wrong with runtime conditionals ? :-(

Anyway, do you get tot he actual call to the dll ? Debug till you get there and check the paramters and then step over it.

[Astro]

push 13
push offset HashMe
push offset Digest
call hFunc ; virtual void CalculateDigest (byte *digest, const byte *input, size_t length)
; 2021  7E4 000283F0 ?CalculateDigest@HashTransformation@CryptoPP@@UAEXPAEPBEI@Z

Best regards,
Robin.

[BlackVortex]

I stepped through it ... I think it uses FASTCALL calling convention.

Can anyone help with that ?

[Astro]

Interesting....... OK.

I'm still a bit blind when reading code I didn't write, but I'm getting there slowly.

I did find a resource that said how to read decorated C++ export names. I'll see if I can find it again. It explained what all the stuff at the end meant. Might offer some clues.

Thanks!

Best regards,
Robin.