Need GZIP inflate help

[Proto]

I've been attempting to write an app that would perform basic interaction with servers via http using the wininet api. I've hit a road block, I can't figure out how to decompress (inflate) the returned data when I use "Accept-Encoding: gzip, deflate". (Wininet does support gzip decompression, but only in Vista - which is a problem)

The simplest solution would be to use gzip.exe and pass command line arguments, but I am unwilling to involve filesystem read/write api in a lightweight, fast application (thats what I'm aiming for at least). In the only reasonably similar thread someone else suggested to use zlib, and I've spent the last 3-4 hours attempting to figure out how to use zlib. There are no includes for MASM, so I figured using LoadLibrary and GetProcAddress would work.

Unfortunately, the required structs for gzip inflation using zlib are confusing, and exactly what the calling application is required to do is also confusing. I would really appreciate if anyone can help, even if its in the form of a simpler but still efficient alternate method.

EDIT :  (Resolved)

This is not a help desk, it is a forum of members.

[Biterider]

Hi
I'm not sure if this lib fits your needs, but give it a try

http://www.mikekohn.net/file_formats/kunzip.php

Regards,

Biterider

[Proto]

Okay, heres what I've got so far:

Code Select Expand

Z_STREAM_S STRUCT
    next_in         DWORD ?
    avail_in        DWORD ?
    total_in        DWORD ?
    next_out        DWORD ?
    avail_out       DWORD ?
    total_out       DWORD ?
    msg             DWORD ?
    state           DWORD ?         ;struct internal_state FAR *state; /* not visible by applications */
    zalloc          DWORD ?         ;alloc_func zalloc;  /* used to allocate the internal state */
    zfree           DWORD ?         ;free_func  zfree;   /* used to free the internal state */
    opaque          DWORD ?         ;voidpf     opaque;  /* private data object passed to zalloc and zfree */
    data_type       DWORD ?
    adler           DWORD ?
    reserved        DWORD ?
Z_STREAM_S ENDS

Code Select Expand

.data
strzlib db "zlib1.dll",0
strInfi db "inflateInit2_",0
hInfi   dd 0
strInf  db "inflate",0
hInf    dd 0
strInfe db "inflateEnd",0
hInfe   dd 0
hzdll   dd 0

Code Select Expand


;setup
invoke LoadLibrary,addr strzlib
mov hzdll,eax
invoke GetProcAddress,hzdll,addr strInfi
mov hInfi,eax
invoke GetProcAddress,hzdll,addr strInf
mov hInf,eax
invoke GetProcAddress,hzdll,addr strInfe
mov hInfe,eax

Code Select Expand

gzip proc   pIn:DWORD,fIn:DWORD,pOut:DWORD,fOut:DWORD
local stream:Z_STREAM_S

mov stream.zalloc,0
mov stream.zfree,0
mov stream.opaque,0

push pIn
push fIn
push pOut
push fOut
pop stream.avail_out
pop stream.next_out
pop stream.avail_in
pop stream.next_in

push 15d or 32d
push stream
call hInfi
push 4h
push stream
call hInf
push stream
call hInfe

ret
gzip endp

At this point I'd just like my application to not crash. Zlib lacks includes/lib files for MASM, and I'm a little out of my depth when it comes to acquiring fuction addresses and the like. I get the error 'the instruction at 0x10000557f referenced memory at 0xf2fb01b0, the memory cannot be read". This appears to happen when calling inflateInit2_.

Zdll is supposed to be able to handle corrupted input data, so I think that the problem might actually be with my code for acquiring and utilising the exports from zdll.

[fearless]

Ive used zlib in the past for uncompressing files compressed with gzip format and found no include files for it as well.

With the zlib compiled DLL version found at:

http://www.zlib.net/zlib123-dll.zip

it includes the zlib1.dll and zlib.lib

Here is my partial attempt at creating a basic .inc file for use in my own little project:

Code Select Expand


; ZLIB include file for Win32 Asm
;
; Extracts taken from zlib.h header file and converted for Win32 asm prototypes
;
; KSR 02/10/2006
;


; Return codes for the compression/decompression functions. Negative
; values are errors, positive values are used for special but normal events.
;

Z_OK            equ 0
Z_STREAM_END    equ 1
Z_NEED_DICT     equ 2
Z_ERRNO         equ -1
Z_STREAM_ERROR  equ -2
Z_DATA_ERROR    equ -3
Z_MEM_ERROR     equ -4
Z_BUF_ERROR     equ -5
Z_VERSION_ERROR equ -6


; Prototype declarations
;

;uncompress PROTO :WORD, :DWORD, DWORD, :DWORD

uncompress PROTO C dest:PTR DWORD, destLen:PTR DWORD, source:PTR DWORD, sourceLen:PTR DWORD


zlibVersion PROTO C  
; On return, EAX points to string containing version of ZLIB library in use


I tested the zlibVersion to see if i was on the right track - that worked ok

Code Select Expand

Invoke zlibVersion
mov edx, eax
Invoke dwtoa, edx, Addr SmallBuffer
Invoke MessageBox, hWin, edx, CTEXT("Zlib Version"),MB_OK


I only had to use the uncompress function and so didnt look into any other, and it was uncompressing Baldurs Gate BIF (gzip format) files this roughly how i used it:

Code Select Expand


Invoke GetFileSize, hFile, NULL
mov ldwFileSize, eax
add eax, 4 ; add 4 because start offset of 1 dword will contain file memory length

;About to allocate memory

Invoke GlobalAlloc, GMEM_MOVEABLE+GMEM_DISCARDABLE, eax
.IF eax != NULL
mov hMemSrc, eax
Invoke GlobalLock, hMemSrc

.IF eax != NULL
; Read file contents into allocated memory region
mov mSrcBIF, eax
;PrintText 'BIFUncompres:BIFReadEntireFile::Call'
Invoke BIFReadEntireFile, hFile, mSrcBIF, IE_READ_FILE
.IF eax == TRUE
;PrintText 'BIFUncompress:BIFReadEntireFile::Success'

; Call functions to calculate the size of mem to allocate to uncompressed file
Invoke BIFCompressedSize, hFile, Addr ldwCompressedSize, AccessType
Invoke BIFUncompressedSize,hFile, Addr ldwUncompressedSize, AccessType
Invoke BIFCompressedData,hFile, Addr ldwCompressedData, AccessType

mov eax, ldwUncompressedSize
add eax, 4 ; skip file memory length
Invoke GlobalAlloc, GMEM_MOVEABLE+GMEM_DISCARDABLE, eax
.IF eax != NULL
mov hMemDest, eax
Invoke GlobalLock, hMemDest
.IF eax != NULL
mov mDestBIF, eax ; save handle to memory area

; Destination - handle to allocated memory to contain uncompressed data
mov ebx, mDestBIF
add ebx, 4 ; skip file memory length
mov dest, ebx

; Source - handle of compressed data in memory
mov ebx, mSrcBIF
add ebx, 4 ; skip file memory length
add ebx, ldwCompressedData
mov src, ebx

;PrintText 'UncompressedData Offset'

; Destination length - uncompressed data size
mov ebx, ldwUncompressedSize
mov destLen, ebx

; Source length - compressed data size
mov ebx, ldwCompressedSize
mov srcLen, ebx

PrintText 'BIFUncompress::ZLIB->Uncompress::Call'

Invoke uncompress, dest, Addr destLen, src, srcLen

.IF eax == Z_OK ; ok
PrintText 'BIFUncompress::ZLIB->Uncompress::Success'


Which all worked ok. Similarly the compress function works the same way - although i seem to have lost the code for that example.

not sure if this will help you, it may be a step in the right direction, or might be of no use to your project.

Based on the InflateInit2 function you are looking to use, in the zlib.h file (which contains most of the documentation - which you probably already know im sure) it says:

Code Select Expand

ZEXTERN int ZEXPORT inflateInit2 OF((z_streamp strm, int  windowBits));


This is another version of inflateInit with an extra parameter. The fields next_in, avail_in, zalloc, zfree and opaque must be initialized before by the caller.

Might be a worthy project for figure out a proper zlib.inc file for use with masm. If you do get anywhere with the inflate functions using the data streams then hopefully you can post here and let us know how you got on.

Best of luck.

[Proto]

Well, now I've got zlib being loaded correctly and the functions I want appear to be loaded successfully.

Code Select Expand

gzip proc   pIn:DWORD,fIn:DWORD,pOut:DWORD,fOut:DWORD
local stream:Z_STREAM_S

mov stream.zalloc,0
mov stream.zfree,0
mov stream.opaque,0

push pIn
push fIn
push pOut
push fOut
pop stream.avail_out
pop stream.next_out
pop stream.avail_in
pop stream.next_in

invoke inflateInit2_,addr stream,15+32
invoke inflate,addr stream,4h
invoke inflateEnd,addr stream

ret
gzip endp

Only problem is that I'm getting Z_VERSION_ERROR for inflateInit2_, followed by a Z_STREAM_ERROR for inflate and inflateEnd. I think it might have something to do with my poor job of translating the Z_STREAM_S struct, but overall I'm not sure.

Edit: Apparently inflateInit2 is actually just a wrapper for inflateInit2_, which has two extra arguments. Only now, after passing the correct arguments and getting Z_OK returned, calling inflate results in Z_STREAM_ERROR and only like the first 30-40 bytes correctly decompressed/inflated.

[Proto]

Right, finally got it working. Thanks to KSR for his includes file. For the benefit of whoever else needs GZIP inflate via zlib:

Code Select Expand

; ZLIB include file for Win32 Asm
;
; Extracts taken from zlib.h header file and converted for Win32 asm prototypes
;
; KSR 02/10/2006 - and contributions from Proto 25/4/2008
;


; Return codes for the compression/decompression functions. Negative
; values are errors, positive values are used for special but normal events.
;


Z_OK            equ 0
Z_STREAM_END    equ 1
Z_NEED_DICT     equ 2
Z_ERRNO         equ -1
Z_STREAM_ERROR  equ -2
Z_DATA_ERROR    equ -3
Z_MEM_ERROR     equ -4
Z_BUF_ERROR     equ -5
Z_VERSION_ERROR equ -6


Z_STREAM_S STRUCT
    next_in         DWORD ?
    avail_in        DWORD ?
    total_in        DWORD ?
    next_out        DWORD ?
    avail_out       DWORD ?
    total_out       DWORD ?
    msg             DWORD ?
    state           DWORD ?   
    zalloc          DWORD ? 
    zfree           DWORD ?
    opaque          DWORD ?       
    data_type       DWORD ?
    adler           DWORD ?
    reserved1        DWORD ?
Z_STREAM_S ENDS


; Prototype declarations
;

;uncompress PROTO :WORD, :DWORD, DWORD, :DWORD

uncompress   PROTO C  dest:PTR DWORD, destLen:PTR DWORD, source:PTR DWORD, sourceLen:PTR DWORD



inflateInit2_ PROTO C :DWORD,:DWORD,:DWORD,:DWORD
inflate PROTO C :DWORD,:DWORD
inflateEnd PROTO C :DWORD

;inflateInit2 (gzip inflate) calls inflateInit2_ with the addr of the zlibversion string and the size of Z_STREAM_S as 3rd and 4th args


zlibVersion PROTO C 
; On return, EAX points to string containing version of ZLIB library in use

Code Select Expand

.data
zlibv db "1.2.3",0    ;zlib version string, pass as 3rd arg to inflateInit2_

Code Select Expand

gzipInflate proc   pIn:DWORD,fIn:DWORD,pOut:DWORD,fOut:DWORD
local stream:Z_STREAM_S

;pIn = pointer to input buffer, fIn = number of bytes in input buffer
;pOut = pointer to output buffer, fOut = number of bytes of free space in output buffer
;function returns the number of bytes proccessed ie. fOut(original) - fOut(new)

mov stream.zalloc,0
mov stream.zfree,0
mov stream.opaque,0
mov stream.total_out,0

mov eax,pIn
mov stream.next_in,eax
mov eax,fIn
mov stream.avail_in,eax
mov eax,pOut
mov stream.next_out,eax
mov eax,fOut
mov stream.avail_out,eax

invoke inflateInit2_,addr stream,47d,addr zlibv,14*4   ;14*4 ;SIZEOF stream
@@:
invoke inflate,addr stream,4h
test eax,eax
jz @b
invoke inflateEnd,addr stream
mov eax,stream.total_out

ret
gzipInflate endp


(I couldn't figure out how to place the zlib version string in the includes file, and I didn't want an extra call (zlibVersion) so I placed it in the data section.)

[fearless]

Nicely done Proto. Glad i could be of help.

[themaster]

Maybe somebody has the same thing done for x64 code? With all addresses 64 bits length, and so on? Or nobody did it yet, and I have to do it by myself?