Reading full response from InternetReadFile into one buffer?

[xxxxxx]

I hope my question is not too dumb, I've already asked in a few places and no one responded.

I am trying to retrieve a website content into a buffer using wininet, the http request works fine and the page is downloaded, but it's downloaded in "chunks", and I want to have the full response inside one buffer.

The code is something like this:

Code Select Expand

.data?
Buffer db 65536 dup(?)
Bufferlen dd ?


------


getpage:
    invoke InternetReadFile, hRequest, addr Buffer, sizeof Buffer, addr Bufferlen
    invoke StdOut, offset lpBuffer ;  <- Buffer contains chunk of response
    .if Bufferlen != 0
        jmp short getpage
    .endif

invoke StdOut, offset Buffer ; <- Buffer contains last chunk of response

------

In C++ I would read the entire response into the same buffer with something like this:

Code Select Expand

int BufferPointer = 0;

while(true) {
----------
InternetReadFile(hRequest, Buffer+BufferPointer, sizeof(Buffer), &readBytes);
BufferPointer += readBytes;
if(readBytes==0) break;
----------
}

// Buffer contains full response


How can I do the same in masm without having to copy each chunk of the response into another buffer?

[qWord]

hi,
you can do it exact the same way as in c++. You may use ESI or EDI as pointer to the buffer, which is summed up with readBytes each iteration.
However, you know that you code can produce an buffer overflow?

[xxxxxx]

Yea I am aware of the BO, but the code is not for production so I don't really care :P

I will try using ESI/EDI, thanks.

[xxxxxx]

Code Select Expand

lea esi, Buffer

read:
    invoke InternetReadFile, hRequest, esi, sizeof Buffer, addr Bufferlen
    add esi, Bufferlen
    .if Bufferlen != 0
        jmp short read
    .endif

Worked fine only when debugging and setting breakpoints in the loop, I then tested adding a sleep in the loop and it does retrieve the full response, otherwise it doesn't

Why is this happening? Why do I need to add a sleep?

[qWord]

To ensure all data is retrieved, an application must continue to call the InternetReadFile function until the function returns TRUE and the lpdwNumberOfBytesRead parameter equals zero

[xxxxxx]

I added

Code Select Expand

.if Bufferlen != 0 || eax == TRUE

But it will still not retrieve the full response, only when the timer (of at least 200ms) is placed into the loop.

[qWord]

maybe this does it:

Code Select Expand

mov ebx,sizeof Buffer
.while 1
    .break .if !rv(InternetReadFile, hRequest, esi, ebx, addr Bufferlen) || Bufferlen == 0
    add esi, Bufferlen
    sub ebx, Bufferlen
.endw

[clive]

the http request works fine and the page is downloaded, but it's downloaded in "chunks"

That's because the internet sends data in packets, the web server may not be sending it as a single piece and the delay between packets isn't predictable, or for that matter guaranteed to be delivered. The MTU of nodes on the way will also cause packets to be fragmented. Wired into the office ethernet things can appear reasonably predictable, but attach to a wireless GSM network and you have to consider all the potential failure conditions. ie If you create an application with dreamy assumptions it will fail when others try to use it in less than ideal conditions, and even marginally impaired ones.

You either make a big buffer and accumulate content, or you process it a bit at a time as it arrives, and use a state machine to determine if you are parsing header data, or content. It's not simple, but is the reality of most applications.