MD5 Hash using Crypt API

[donkey]

This snippet is from WinExplorer but I thought it would be useful to post as a separate thread as some people have emailed me asking how I created the hash function. It was originally from a C example in the MSDN library translated to GoAsm syntax. It requires a fully qualified path to a file to generate a hash for and a pointer to a string to hold the resulting hash in HEX (33 bytes for MD5, 41 bytes for SHA) and an algorithm ID, for MD5 set dwHashType to CALG_MD5.

Code Select Expand

GetHash FRAME pszFilename, pHashString, dwHashType
uses edi,esi,ebx
LOCAL dwStatus :D
LOCAL bResult :D
LOCAL hProv :D
LOCAL hHash :D
LOCAL hFile :D
LOCAL cbHigh :D
LOCAL prgbFile :D
LOCAL cbRead :D
LOCAL rgbHash[64] :B
LOCAL cbHash :D
LOCAL rgbDigits[16] :B

/*
dwHashType is one of the following CALG_XXX hash types

#define CALG_MD2  08001h
#define CALG_MD4  08002h
#define CALG_MD5  08003h
#define CALG_SHA  08004h
#define CALG_SHA1 08004h ; (Same as CALG_SHA)
#define CALG_MAC  08005h
*/

// Create a 1MB buffer to hold the file data during the hash
// Note that this is just a buffer for block reads and does not limit file size
invoke VirtualAlloc,NULL,000100000h,MEM_COMMIT,PAGE_READWRITE
mov [prgbFile],eax
test eax,eax
jz >>.ERRORMEM

mov [rgbDigits],"0123"
mov [rgbDigits+4],"4567"
mov [rgbDigits+8],"89ab"
mov [rgbDigits+12],"cdef"

invoke CreateFile,[pszFilename],GENERIC_READ,FILE_SHARE_READ,NULL,\
OPEN_EXISTING,FILE_FLAG_SEQUENTIAL_SCAN,NULL
mov [hFile],eax
test eax,eax
js >>.ERRORFILE

invoke GetFileSize,[hFile],offset cbHigh
cmp D[cbHigh],0
jne >
test eax,eax
jz >>.ERRORCONTEXT
:

invoke CryptAcquireContext,offset hProv,NULL,NULL,PROV_RSA_FULL,CRYPT_VERIFYCONTEXT
test eax,eax
jz >>.ERRORCONTEXT

invoke CryptCreateHash,[hProv], [dwHashType], NULL, NULL, offset hHash
test eax,eax
jz >>.ERRORHASH

:
invoke ReadFile, [hFile], [prgbFile], 000100000h, offset cbRead, NULL
test eax,eax
jz >>.ERROROTHER
mov [bResult],eax
cmp D[cbRead],NULL
je >.ENDOFFILE
invoke CryptHashData,[hHash], [prgbFile], [cbRead], NULL
test eax,eax
jz >>.ERROROTHER
jmp <
.ENDOFFILE

mov D[cbHash],64
invoke CryptGetHashParam,[hHash], HP_HASHVAL, offset rgbHash, offset cbHash, NULL
test eax,eax
jz >.ERROROTHER

// convert the hash to an SHA string using a lookup table
xor eax,eax
xor edx,edx
mov ebx,[pHashString]
mov edi,offset rgbHash
mov ecx,[cbHash]
mov esi,offset rgbDigits
:
mov al,[edi]
shr al,4
mov dl,[esi+eax]
mov [ebx],dl
inc ebx
mov al,[edi]
and al,0fh
mov dl,[esi+eax]
mov [ebx],dl
inc edi
inc ebx
dec ecx
jnz <
mov B[ebx],0

invoke CryptDestroyHash [hHash]
invoke CryptReleaseContext, [hProv], NULL
invoke CloseHandle,[hFile]
invoke VirtualFree,[prgbFile],NULL,MEM_RELEASE

xor eax,eax ; Return success (0)
RET

.ERROROTHER
invoke CryptDestroyHash [hHash]

.ERRORHASH
invoke CryptReleaseContext, [hProv], NULL

.ERRORCONTEXT
invoke CloseHandle,[hFile]

.ERRORFILE
invoke VirtualFree,[prgbFile],NULL,MEM_RELEASE

.ERRORMEM
xor eax,eax
dec eax ; Return error (-1)
RET

ENDF

[yida]

Hi, Donkey:
I'm trying to convert this into a MASM grammar and having few questions as below:
1.For your code lines:
mov [rgbDigits],"0123"
mov [rgbDigits+4],"4567"
mov [rgbDigits+8],"89ab"
mov [rgbDigits+12],"cdef"
I've written something like this:
mov dword ptr rgbDigits[1], '0123'
mov dword ptr rgbDigits[5], '4567'
mov dword ptr rgbDigits[9], '89ab'
mov dword ptr rgbDigits[13], 'cdef'
Is this correct? I didn't get any compilation error, but I'm not sure if this will work or not. I know this is to put each character into the array so that from the first element of the array to the last element, it's '01234567890abcdef'.
2.For your code lines where it "convert the hash to an SHA string using a lookup table", I've written something like this:

;Convert the hash value to an SHA string.
xor eax,eax
xor edx,edx
mov ebx,szHashString
mov edi,addr @rgbHash
mov ecx,@cbHash
mov esi,prt addr @rgbDigits
looping:
mov al,[edi]
shr al,4
mov dl,[esi+eax]
mov bl,dl
inc ebx
mov al,[edi]
and al,0fh
mov dl,[esi+eax]
mov bl,dl
inc edi
inc ebx
dec ecx
jnz looping
mov ebx,0
These lines did not pass the compilation, complaining "invalid instruction operands" in the following two lines:
mov edi,addr @rgbHash
mov esi,prt addr @rgbDigits
Also, I've modified your code 'mov [ebx],dl' into 'mov bl,dl'. I think this is wrong, too.
Could you tell me how to correct them?
And plus, I've used addr instead of offset in crypto API calls, am I right?
I'm new to 32-bit assembly in MASM and GOASM. please forgive my asking this question as it sounds so simple to you. Thanks very much

Yida

[donkey]

When moving a literal string as a DWORD value GoAsm reverses the string for you so in MASM mov [rgbDigits],"0123" would be mov DWORD PTR [rgbDigits],"3210", alternatively you can use mov [rgbDigits], 33323130h. Remember that the x86 stores DWORDs in little endian format, so the byte order must be reversed. The sequence you posted simply builds a lookup table that holds "0123456789ABCDEF", I did it that way because I wanted it to be held on the stack.

Are you allowed to start a data label with @ in MASM ? been quite a long time since I used it.

[yida]

Yeah ,sure. A variable stared with a "@" is allowed but not re commented.
Thank you so much for responding so fast. I understand this one then.
How about others? Thanks.

Yida

[donkey]

mov [ebx],dl

mov the value in Dl to the address stored in EBX

mov bl,dl

mov the value in dl into the register bl

they are completely different things, the syntax however is the same in both MASM and GoAsm so there is no need to modify the original line.

Yes, MASM differentiates between OFFSET and ADDR, OFFSET is used for the data and code segments, ADDR for the stack segment. You have to use ADDR for local variables because of the way MASM calculates the offsets, in GoAsm they are interchangeable so I tend to use only OFFSET.

mov esi,prt addr @rgbDigits

PTR is spelled wrong but that isn't why you have an error, becuase of the way local offsets are calculated (the value of ESP is not known at assembly time) you cannot move an address of a LOCAL directly into a register, try

lea edi,@rgbHash
lea esi,@rgbDigit

This is a limitation of the x86 family of processors, not any particular assembler, GoAsm changes the line to an LEA calculation at assembly time.

[yida]

OK. Got it.
Thank you very much for the help. I've passed the compilation now.

[yida]

Hi, Mr. Donkey:
I have successfully converted your function into MASM grammer and was able to create a dll file containing this function.
The following is my code:
MD5HASHFILE PROC szFileName :LPSTR, szHashString :LPSTR, dwHashType :DWORD

   local @hFile
   local @dwFileSizeHigh
   local @dwFileSizeLow
   local @cbRead
   local @bResult
   local @hProv
   local @hHash
   local @prgbFile
   local @rgbHash[64]:byte
   local @cbHash
   local @rgbDigits[16]:byte
   
   ;initialize array
   mov DWORD PTR [@rgbDigits],"3210"
   mov DWORD PTR [@rgbDigits+4],"7654"
   mov DWORD PTR [@rgbDigits+8],"ba98"
   mov DWORD PTR [@rgbDigits+12],"fedc"

   ;Create a 1MB buffer to hold the file data during the hash
   ;Note that this is just a buffer for block reads and does not limit file size
   invoke VirtualAlloc,NULL,dwBufferSize,MEM_COMMIT,PAGE_READWRITE
   .if eax
      mov @prgbFile,eax
   .else
      jmp ERRORMEM
   .endif
   ;PrintStringByAddr szFileName
   ;invoke   MessageBoxW,NULL,szFileName,addr szError,MB_OK or MB_ICONINFORMATION
   
   ;Open the file
   invoke   CreateFileW, szFileName,GENERIC_READ,FILE_SHARE_READ,0,\
   OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0
   .if   eax ==   INVALID_HANDLE_VALUE
      invoke   MessageBox,hInstance,addr szErrOpenFile,NULL,MB_OK or MB_ICONEXCLAMATION
      jmp ERRORFILE
   .else
      mov @hFile,eax
   .endif
   invoke   GetFileSize,@hFile,addr @dwFileSizeHigh
   cmp @dwFileSizeHigh, 0
   jl ERRORCONTEXT
   mov   @dwFileSizeLow,eax
   cmp eax, 0
   jl ERRORCONTEXT
   
   ;Prepair Hash context.
   invoke CryptAcquireContext,addr @hProv,NULL,NULL,PROV_RSA_FULL,CRYPT_VERIFYCONTEXT
   cmp eax, 0
   jle ERRORCONTEXT
   invoke CryptCreateHash,@hProv, CALG_MD5, NULL, NULL, addr @hHash
   cmp eax, 0
   jle ERRORCONTEXT
   
   ;Read file and Do the hashing.
   reading:
   invoke ReadFile, @hFile, @prgbFile, dwBufferSize, addr @cbRead, NULL
   cmp eax,0
   jl ERROROTHER
   mov [@bResult],eax
   cmp [@cbRead],NULL
   je EOF
   invoke CryptHashData, @hHash,@prgbFile,@cbRead,NULL
   cmp eax,0
   jl ERRORHASH
   jmp reading
   ;Get the hash value.
   EOF:
   mov [@cbHash], 64
   invoke CryptGetHashParam,@hHash, HP_HASHVAL, addr @rgbHash, addr @cbHash, NULL
   cmp eax,0
   jle ERROROTHER
   ;Convert the hash value to an SHA string.
   xor eax,eax
   xor edx,edx
   mov ebx,szHashString
   lea edi,@rgbHash
   mov ecx,[@cbHash]
   lea esi,@rgbDigits
   looping:
   mov al,[edi]
   shr al,4
   mov dl,[esi+eax]
   mov [ebx],dl
   inc ebx
   mov al,[edi]
   and al,0fh
   mov dl,[esi+eax]
   mov [ebx],dl
   inc edi
   inc ebx
   dec ecx
   jnz looping
   mov ax,0
   mov [ebx],ax
   
   ;Release crypt context.
   invoke CryptDestroyHash, @hHash
   invoke CryptReleaseContext, @hProv, NULL
   ;Release file.
   invoke CloseHandle,@hFile
   ;Release memory buffer
   invoke VirtualFree,@prgbFile,NULL,MEM_RELEASE
   
   ending:
   xor eax,eax
   ret
   
   ERROROTHER:
   invoke CryptDestroyHash, @hHash
   
   ERRORHASH:
   invoke CryptReleaseContext, @hProv, NULL
   
   ERRORCONTEXT:
   invoke CloseHandle,@hFile
   
   ERRORFILE:
   invoke VirtualFree,@prgbFile,NULL,MEM_RELEASE
   
   ERRORMEM:
   xor eax,eax
   dec eax ; Return error (-1)
   RET
   
MD5HASHFILE Endp
I'm using this dll as a function library and calling this MD5HASHFILE function from PowerBuilder 11.2 program. The problem there was the "Unicode".
As you can see that I'm using CreateFileW instead of CreateFile while openning the file. PowerBuilder defaults all strings it processes as UTF-16LE strings. This lead to the fact that when getting back the szHashString as the result, it automatically extends szHashString into an UTF-16LE string and thus resulted in variable out of boundary, which screwed the whole programm.
I'm thinking of converting the string szHashString to a UTF-16LE string inside the dll, or just make the function return the hash result string pointer instead of modifying it in the dll as an output parameter.
But the problem is that I don't know how to do them. Could you please point me a direction in doing these? Thanks.

[donkey]

Hi yida,

A UTF16LE string which I believe is the same as a Unicode BSTR string can be made as follows:

Code Select Expand

invoke MultiByteToWideChar, CP_ACP, NULL, offset wszResult, -1, offset szAnsiString, LENGTH
invoke SysAllocString, offset wszResult
mov [pbszString],eax

You must free the resulting string when you are done with it using

Code Select Expand

invoke SysFreeString,[pbszString]

For the reverse you can convert the BSTR to ANSI as follows:

Code Select Expand

invoke WideCharToMultiByte, CP_ACP, NULL, [pbszString], -1, offset szResult,, LENGTH, "", NULL

[yida]

Hi, Donkey:
I've declared another variable(@szHashResult) to hold the ANSI hash result string. I've also declared another variable (@szHashResultWide) to hold the Unicode string.And then I coded the converting part as follows:

Code Select Expand


invoke MultiByteToWideChar, CP_ACP, NULL, addr @szHashResultWide, -1, addr @szHashResult, LENGTHOF @szHashResult
invoke SysAllocString, addr @szHashResultWide
mov [szHashString],eax
invoke SysFreeString,[@szHashResultWide]

The result in szHashString was 3 wired charactors and my test program crashes for the reason "access violation", the ip pointer was pointed to some blank area.
What was the root cause of this and how can I resolve it? Thanks.

[yida]

And the content of @szHashResultWide was 3 wired characters after calling the api function "MultiByteToWideChar". Could there be a problem in the API call "MultiByteToWideChar"?

[yida]

Part of the reason was my declaration, I've changed the declaration of the unicode string into

Code Select Expand

local @szHashResultWide[32]:byte
And the API call should look like this:

Code Select Expand


invoke MultiByteToWideChar, CP_ACP, NULL, addr @szHashResult, -1, addr @szHashResultWide, 21h

Which means the ansi string should be put first.
But still, problem here. I can get the full hash result string in the @szHashResultWide now. But after the API call, some other declared string was changed into something not readable. Like I've declared a global (global to the whole dll file) variable like this:

Code Select Expand

szError db "Error",0
I use this variable to fill the caption of my message box. After the API call it was changed to something not readable. I think the API call had made some variables out of boundary. But still, why, and how to fix it?
Thanks.

[yida]

Hi, Donkey:
Problem solved.
The root cause was that in PowerBuilder program, I didn't initialize the string the dll file was to return. This made the string out of boundary in Powerbuilder program and crashed it.
The "wired characters" in message box was because of "Unicode". The function MessagBoxW expects unicode string as the title and text, while the title I declared was ANSI.
I did not use the api functions you told me to get a unicode result. Instead, I changed your "convert hash value" part as follows:

Code Select Expand


;Convert the hash value to an SHA string.
xor eax,eax
xor edx,edx
mov ebx,szHashString
lea edi,@rgbHash
mov ecx,[@cbHash]
lea esi,@rgbDigits
looping:
mov al,[edi]
shr al,4
mov dl,[esi+eax]
mov [ebx],dl
inc ebx
mov dl,0
mov [ebx],dl
inc ebx
mov al,[edi]
and al,0fh
mov dl,[esi+eax]
mov [ebx],dl
inc edi
inc ebx
mov dl,0
mov [ebx],dl
inc ebx
dec ecx
jnz looping
mov ax,0
mov [ebx],ax

It worked out fine. And also, I've created an unicode version of this function in my dll. After this, I've got 2 functions as for ANSI and UNICODE, just like most windows API string related functions.
Well, this is my first MASM program that could be put into use as a real solution. I would like to take this chance to say Thank You to you, as your help and supporting was the biggest motivation upon my finishing this task. And I also want to say sorry about my bothering you with so many posts as asking very simple questions. Hope you well.

Regards,
Yida

[donkey]

Glad you worked it out Yida, sorry I haven't been responding but I've been pretty busy with my Help2 viewer.