fpu problem

[minor28]

To change floating numbers for example st(0)=59.433333333 to st(0)=59.433330000 I multiply with 10000, load the control word to truncate and then divide with 10000.

Code Select Expand


fld fValue ;59.433333333
mov integer,10000
fimul integer
fstcw cw
movzx eax,cw
and ax,1111001111111111b
add ax,0000110000000000b ;truncate
mov cw,ax
fldcw cw
frndint
fidiv integer ;=> 5.9433299999999995e+0001


But this will not give the exact result. How do I do this operation?

[jj2007]

What do you get if you put finit in front? Or if you use:

Code Select Expand

and ax, 1111000011111111b
or ax, 0000111100000000b ;truncate, 64 bits precision

I suggest to check it with Olly, step by step. You will find out that precision counts :bg

[minor28]

Each one of your suggestions did what I wanted. I should have thought of finit from the beginning.

Thanks

[dedndave]

it may be best to perform all intermediate calculations at high precision
then, when the numbers are presented to the user, tailor the text strings to your needs, rather than the floats
as you have found out, there may be no efficient way to precisely represent most values in binary form

[minor28]

Yes, if the result is presented to the user. In this case it is a part of a more extensive calculation.

[ToutEnMasm]

Perhaps this can help

http://www.masm32.com/board/index.php?topic=11749.0

[dedndave]

Yes, if the result is presented to the user. In this case it is a part of a more extensive calculation.

in that case, you may want to keep all the digits