Hi Everyone,
Can anyone verify the results of calculation below ? var n1:Number=18.9; var n2:Number=100.0; var n3:Number=n1*n2; //1889.9999999999998 is it bug ? thanks spiros |
The Number data type, when including a decimal point is a binary floating point number and binary floating point math can result in rounding errors like yours.
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680 <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680> Adobe says this about Number: "The Number data type uses the 64-bit double-precision format as specified by the IEEE Standard for Binary Floating-Point Arithmetic (IEEE-754) which means rounding errors can occur... The Number data type uses 52 bits to store the significand, with the result that numbers that require more than 52 bits to represent precisely, such as the fraction 1/3, are only approximations. If your application requires absolute precision with decimal numbers, you need to use software that implements decimal floating-point arithmetic as opposed to binary floating-point arithmetic." Erik On Feb 2, 2018, at 7:22 AM, Σπύρος Αγγελόπουλος <[hidden email]> wrote: Hi Everyone, Can anyone verify the results of calculation below ? var n1:Number=18.9; var n2:Number=100.0; var n3:Number=n1*n2; //1889.9999999999998 is it bug ? thanks spiros |
You can sort of fix this issue by rounding, but it just hides the issue and should not be used for financial applications:
var setPrecision:Function = function(number:Number, precision:int) { precision = Math.pow(10, precision); return Math.round(number * precision)/precision; } var number:Number = 10.98813311; trace(setPrecision(number,1)); //Result is 10.9 trace(setPrecision(number,2)); //Result is 10.98 trace(setPrecision(number,3)); //Result is 10.988 and so on https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript <https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript> Erik On Feb 2, 2018, at 8:39 AM, Erik J. Thomas <[hidden email]> wrote: The Number data type, when including a decimal point is a binary floating point number and binary floating point math can result in rounding errors like yours. https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680 <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680> Adobe says this about Number: "The Number data type uses the 64-bit double-precision format as specified by the IEEE Standard for Binary Floating-Point Arithmetic (IEEE-754) which means rounding errors can occur... The Number data type uses 52 bits to store the significand, with the result that numbers that require more than 52 bits to represent precisely, such as the fraction 1/3, are only approximations. If your application requires absolute precision with decimal numbers, you need to use software that implements decimal floating-point arithmetic as opposed to binary floating-point arithmetic." Erik On Feb 2, 2018, at 7:22 AM, Σπύρος Αγγελόπουλος <[hidden email]> wrote: Hi Everyone, Can anyone verify the results of calculation below ? var n1:Number=18.9; var n2:Number=100.0; var n3:Number=n1*n2; //1889.9999999999998 is it bug ? thanks spiros |
Eric,
Thanks for the quick response. Spiros -----Original Message----- From: Erik J. Thomas [mailto:[hidden email]] Sent: Friday, February 2, 2018 6:49 PM To: [hidden email] Subject: Re: Number Multiplication AS3 You can sort of fix this issue by rounding, but it just hides the issue and should not be used for financial applications: var setPrecision:Function = function(number:Number, precision:int) { precision = Math.pow(10, precision); return Math.round(number * precision)/precision; } var number:Number = 10.98813311; trace(setPrecision(number,1)); //Result is 10.9 trace(setPrecision(number,2)); //Result is 10.98 trace(setPrecision(number,3)); //Result is 10.988 and so on https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript <https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript> Erik On Feb 2, 2018, at 8:39 AM, Erik J. Thomas <[hidden email]> wrote: The Number data type, when including a decimal point is a binary floating point number and binary floating point math can result in rounding errors like yours. https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680 <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680> Adobe says this about Number: "The Number data type uses the 64-bit double-precision format as specified by the IEEE Standard for Binary Floating-Point Arithmetic (IEEE-754) which means rounding errors can occur... The Number data type uses 52 bits to store the significand, with the result that numbers that require more than 52 bits to represent precisely, such as the fraction 1/3, are only approximations. If your application requires absolute precision with decimal numbers, you need to use software that implements decimal floating-point arithmetic as opposed to binary floating-point arithmetic." Erik On Feb 2, 2018, at 7:22 AM, Σπύρος Αγγελόπουλος <[hidden email]> wrote: Hi Everyone, Can anyone verify the results of calculation below ? var n1:Number=18.9; var n2:Number=100.0; var n3:Number=n1*n2; //1889.9999999999998 is it bug ? thanks spiros |
Free forum by Nabble | Edit this page |