The MASM Forum Archive 2004 to 2012

Project Support Forums => MASM32 => Topic started by: jj2007 on September 22, 2011, 10:15:30 PM

Title: A glitch in FloatToStr
Post by: jj2007 on September 22, 2011, 10:15:30 PM
Under certain conditions, FloatToStr erroneously returns 0.0xx as follows:

.data
r8n Real8 0.00999999
r8o Real8 0.0099999999
r8p Real8 0.009999999999
r8q Real8 0.00999999999999
r8r Real8 0.0099999999999999


0.009999990     r8n MasmBasic
9.999990e-003   r8n FloatToStr

0.010000000     r8o MasmBasic
0.000000e-003   r8o FloatToStr

0.010000000     r8p MasmBasic
0.000000e-003   r8p FloatToStr

0.010000000     r8q MasmBasic
0.000000e-003   r8q FloatToStr

0.010000000     r8r MasmBasic
0.000000e-003   r8r FloatToStr


FloatToStr2 is not affected by this rounding error. Example attached (for use with the MasmBasic library (http://www.masm32.com/board/index.php?topic=12460)).