The MASM Forum Archive 2004 to 2012

OS: Win XP SP1,SP2
MASM32 v9.0

Writing some infix/postfix interpreter stuff, i got some kind of a bufferoverflow with
this term: 1/11(-1)*3

It turned out that FloatToStr2, used in this kind:

mycallback proc

LOCAL num1_string [20]:BYTE

invoke FloatToStr2,num1,addr num1_string
invoke Output,addr num1_string,0


needs sometimes 21 Bytes insteat of 19
quote from MASMLIB help:

Quote
Comments
The buffer for the string should be at least 19 bytes long as the procedure
always write an 18 byte long string in szDbl. For alignment considerations,
a 20 byte long buffer is recommended.

Here some (quick&dirty) code:



.486
.model flat,stdcall
option casemap:none

include \masm32\include\windows.inc
include \masm32\include\masm32.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc
   
;     libraries
;     ~~~~~~~~~
includelib \masm32\lib\masm32.lib

includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib   

mycallback proto   
; ««««««««««««««««««««««««««««««««««««««««««««««««««
.data
hInstance dd 0
.code
start:     
      invoke GetModuleHandle,NULL
      mov hInstance,eax
      invoke mycallback
      invoke ExitProcess,0

; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««


mycallback proc
LOCAL num1_string [20]:BYTE


.data
  mynum REAL8 -0.0909090909090909116
.code
invoke FloatToStr2,mynum,addr num1_string

ret
mycallback endp
end start


using ollydbg you can step over the mycallback call

00401007  |.  A3 00304000   MOV DWORD PTR DS:[403000],EAX
0040100C  |.  E8 07000000   CALL rechner.00401018
00401011  |.  6A 00         PUSH 0                                   ; /ExitCode = 0


and see, that last byte of the EBP value is changed
(i know, this is because of the proc-macro ;) :

00401018  /$  55            PUSH EBP
00401019  |.  8BEC          MOV EBP,ESP
0040101B  |.  83C4 EC       ADD ESP,-14

....

LEAVE
ret

EBP is pushed onto stack and than LOCAL reserve the 20 bytes buffer, FloatToStr2 writes 21 bytes
to this buffer -> "saved" EBP value is changed ->LEAVE and RET ->EBP changed

I think this is because of (some code from fptoa2.asm)


; Check for a negative number.

    .if (sdword ptr [fpin][4] < 0)
      and byte ptr [fpin][7], 07fh ; change to positive
      mov byte ptr [edi], '-' ; store a minus sign
      inc edi
    .endif
1 sign byte + inserting "0." (2 bytes) + rep movsb (16 bytes)=1+2+16+1=20

.IF (SDWORD PTR ecx <= 0)
        mov word ptr [edi], '.0'  <--- +2 Bytes
add edi, 2
neg ecx
mov al, '0'   <--- +1 Byte
rep stosb
mov ecx, 16
      .ELSE
        rep movsb
        mov byte ptr [edi], '.'
        inc edi
        mov ecx, 16
        sub ecx, [iExp]
      .ENDIF
      rep movsb   <---------+16 Bytes

after all, EDI points to [EDI+20] and


; Clean up and go home.

ftsExit:
    mov byte ptr [edi], 0
writes the 0 "outside" of the buffer.

Of course, using buffers >20 bytes is a work around, but it would be better to fix that bug ;).

   

QuoteOf course, using buffers >20 bytes is a work around, but it would be better to fix that bug ;).

Or use the FpuFLtoA in the fpu.lib. :8)

BTW, initializing a REAL8 with a value containing more than 16 significant digits is not of much use. The excess digits may be used by some ascii-to-fp algos but would be lost anyway when the float is rounded and stored with REAL8 precision.

Raymond

Quote from: raymond on June 05, 2007, 01:04:50 AM
BTW, initializing a REAL8 with a value containing more than 16 significant digits is not of much use. The excess digits may be used by some ascii-to-fp algos but would be lost anyway when the float is rounded and stored with REAL8 precision.
Raymond

This is only a working example to show this bug   :wink. .
I got this "bufferoverflow" in my application during the calculation of this term: 1/11(-1)*3,
and I am using a REAL8/QWORD to store the FPU result and pass the value to the FloatToStr2. -0.0909090909090909116 is the value in ST(0) after 1/11*(-1)  (copied out to the example code from OllyDbg FPU-output). This value is passed (in ST(0)) to my callback procedure, the procedure pop it in a REAL8, call FloatToStr2 and output this string, so this is not my "standard" initialisation value :wink:


 
Off topic: (application is fixed and works fine, but maybe someone is interested in it)
complete code:

mycallback proc

LOCAL num1_string [20]:BYTE
LOCAL num2_string [20]:BYTE
LOCAL num1:QWORD
LOCAL num2:QWORD
;in ST0,ST1 beide Zahlen
;in eax operator

fstp qword ptr [num1]
cmp al,'='
je @f ;bei = nur einen Operator ausgeben
  fst qword ptr [num2]
@@:
fld qword ptr [num1] ;widerherstellen

movzx ebx,al
mov dword ptr[num2_string],ebx
invoke FloatToStr2,num1,addr num1_string

cmp bl,'='
jne @f
  invoke Output,addr num2_string,0
  invoke Output,addr num1_string,0
  invoke Output, addr CRLF,0
  ret
@@: ;ansonsten: x+y

invoke Output,addr num1_string,0 ;x

invoke Output,addr num2_string,0 ;OP
invoke FloatToStr2,num2,addr num2_string
invoke Output,addr num2_string,0 ;y


ret

mycallback endp
Ok - this is some kind of quck&dirty code, but it works fine - it takes in eax a math operator (ascii) and if it is not '=' ->  pop ST(0), output it, output the operator and the second value, then restore the poped value and return. If the operator is '=' -> output only ST(0)


Here the code and the binary. You can enter (first input field) infix-terms like (1.2+(3-4)*123.123) or (second input field) postfix terms like 1 2 + 3 4+ *


[attachment deleted by admin]