Problem with ToString on a float
I have C# code that calls ToString("0.000000") on a float variable (which according to the docs means that its going to be rounded to 6 digits of precision)
I have a case where the value in the float is -14.6511173 and according to the documentation, it should be converted to either "-14.651117" or "-14.651118" (I can't tell from the docs which way negative numbers are supposed to be rounded) but the output I get is (for some reason) "-14.651120". I am on .NET 5.0. Can anyone tell me why I am seeing 20 instead of the expected 17 or 18? If I ask it for another digit in the ToString, I get "-14.6511200" instead of the expected "-14.6511173"
21 Replies
Huh, curious. Even if you tell it to use 7 or 8 zeroes, it still does
20. Rounding error?
Rounding error.
double works as expectedFloats have somewhere between 6 and 9 significant figures of precision, so that's about as expected
Hmm according to https://www.h-schmidt.net/FloatConverter/IEEE754.html, we can get:
* -14.65111827850341796875
* -14.6511173248291015625
* -14.65111637115478515625
So we should be able to do better than "-14.65112000"
interestingly this suggests it shouldn't be losing precision at this point for this value, but i don't know how accurate this website is
hi
immediate solution is just switch to doubles to get more precision
next most pressing question is, why .NET 5 when it's out of support
canton7
REPL Result: Success
Console Output
Compile: 429.308ms | Execution: 30.050ms | React with ❌ to remove this embed.
Yeah it's not the precision of the float, it's something to do with that format
Its legacy code that I don't want to update if I don't have to.
I didn't know there was a difference between e.g. "0.000" and "F3", but apparently they are different! #allow-unsafe-blocks will know what's going on
$help
How to get the best help :catpog:
Make a post in #help or one of the topic channels under Development.
Avoid asking
:catthinking: Can anybody help me?
:catthinking: Has anyone used XYZ?
:catthinking: Why doesn't my code work?
C# is a big area! No one knows they can help unless you tell them about the small area you're trying to work in.
Explain what you are doing, and potentially why for as much context as possible. Avoid screenshots where possible, share code directly in Discord. Type
$code into chat to learn how to post code.
See https://www.nohello.net and https://dontasktoask.com if you want common help chat room etiquette.you lost?
huh
i just don't know why you did that in a random help thread :lul:
ok
As expected, Tanner knew!
tannergooding
Depends on the value
The former is a custom numeric format string and has some historical limits to preserve back compat in UI display
Quoted by
<@660066004059029524> from #allow-unsafe-blocks (click here)
React with ❌ to remove this embed.
and it's not "broken", it's just limited N6 likewise isn't equivalent, it differs in various other cases its because of compat, because its used for UI display probably "easier" to make the change to more digits now, but still potentially problematic