Posted By

godstroke on 12/15/10


Tagged

number decimal String casting conversiton toPrecision


Versions (?)

String to Number conversion generates unexpected decimals, fix by toPrecision


 / Published in: ActionScript 3
 

This seems to be a bug.. or not.. But explains how to fix it anyway.

  1. var a:Number = 0.99;
  2. var s:String = String(a);trace(s); // 0.99
  3. s = String(a/100); trace(s); // 0.009899999999999999 // 8 ?
  4.  
  5. var p:String = "0.99";
  6. trace(Number(p)/100); // 0.009899999999999999 // 8 ?
  7. trace(Number(p)); // 0.99
  8.  
  9. trace(Number(String(Number(p)/100))*100); // 0.9900000000000001 // 1 ?
  10.  
  11.  
  12. trace( Number(Number(p).toPrecision(8))/100 ); // 0.009899999999999999
  13.  
  14. // fix - as I need the first 2 decimals... problem is solved
  15. trace((Number(p)/100).toPrecision(8)) // 0.0099000000
  16.  
  17. var f:String = (Number(p)/100).toPrecision(8);
  18.  
  19. // seems fixed
  20. var fs:Number = Number(f); trace(fs); // 0.0099
  21.  
  22. // but also
  23. trace(fs*100); // 0.9900000000000001

Report this snippet  

You need to login to post a comment.