-
Notifications
You must be signed in to change notification settings - Fork 7.6k
Fixes String(float) issue with Stack Smashing #6138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The proposed change does not fix the general problem. It fixes the specific minimal test case where the float number is 1e32, but all it would take to break it again is to change 1e32 to 1e33. As stated in the original report, a general fix would require a buffer size of about 80 characters plus additional code to limit the precision value. Furthermore, the general problem also exists in the double precision version of the function, where the buffer size would need to be much larger still. All of that is clearly described in the original issue report. |
I see. I tested other implementations of Arduino and it seems that all of them have the same problem. None is user safe. Your propose would be a user safe implementation that makes sure that it won't break, no matter parameters used. It is an edge case testing. |
I had already considered it and just sent a new commit. void setup() {
Serial.begin(115200);
Serial.printf("\nTesting potential issues with Float-String:\n\n");
String s = String(1e32f, 0);
Serial.println(s);
Serial.println(String(-1.0f, 30));
Serial.println(String(0.0f, 31));
Serial.println(String(0.0f, 100));
Serial.println(String(-234223.4f, 32));
}
void loop() {
// put your main code here, to run repeatedly:
}
Output of this sketch:
|
Not crashing is better than crashing, but silently giving wrong answers is not good either. This is a valid single precision floating point number: -340282346638528859811704183484516925440 It is the most negative number than can be exactly represented by a float. It is 40 digits without a decimal point or postdecimal digits. There is no way to represent that in 33 characters without either switching to exponential notation or completely losing the meaning. Unfortunately, the Arduino String Reference does not say what is supposed to happen for valid but inconveniently large numbers. It would be plausible to convert to exponential notation after a certain size, and to truncate the number of decimal places, or to throw a documented exception, but crashing or silently giving blatantly wrong results is not a reasonable behavior. The fundamental problem is that whoever designed String(float, precision) did not think through all the implications. The simplest solution for single floats is to limit the precision to 38 and use a buffer of 80 characters, thus allowing 39 predecimal characters to represent the largest number, 38 postdecimal characters for the smallest number, 1 sign, 1 decimal point, and a null. That uses more RAM, but not that much more RAM, and it works. For double floats the RAM usage is worse, requiring 524 bytes. The other simple solution is to switch to exponential notation above some threshold - but you still have to limit the precision value. Exponential notation does not exactly meet the implied specification - but the spec is so vague that you might get away with it. The real thing that needs to happen is that whoever is responsible for the Arduino specification should address the issue and fully specify the behavior of String({float/double}, precision). |
I did some experiments. The idea is to find out how ESP32 different routines can print a number such as 1/3. First I just used C sprintf(): char fstr[512];
double d = 1/3.0f;
sprintf(fstr, "%70.70f", d);
Serial.printf("\n[%s]\n", fstr); Output is: Then I modified the String::String(double value, unsigned char decimalPlaces) {
init();
char buf[512];
*this = dtostrf(value, (decimalPlaces + 2), decimalPlaces, buf);
} and also changed the dostrf() back to the original ESP32/ESP8266 code. The result of the excution of Serial.println(String((double)1/3.0f, 99)); is Lastly, I tried to print char fstr[512];
double d = -340282346638528859811704183484516925440;
sprintf(fstr, "%70.70f", d);
Serial.printf("\n[%s]\n", fstr);
Serial.println();
Serial.println(String(d, 99)); The output:
I think that C is not Fortran... So maybe, Arduino may never be suited for what you are looking for. |
I finally tried this: double d = 0.333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333; // forcing compiler to reach 1 / 3.0f;
sprintf(fstr, "%70.70f", d);
Serial.println();
Serial.printf("\n[%s]\n", fstr);
Serial.println(String((double)0.333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333, 99));
Serial.println(); Results:
|
I think that given the results of these experiments, the user shall find out a way to achieve the desired outcome when treating floats, double or any sort of precision decimal point numbers. Arduino String is not designed for this purpose. It's just a way to convert different types to the String Class. |
The results from the 1/3 experiments are exactly as expected. In IEEE 754 single precision floating point representation, [0.33333334326744079589843750...] is the closest you can get to the exact value. Single precision floating point is represented by 23 binary fractional bits. Since a binary number has only powers of 2, a number that has a factor of 3 cannot be represented exactly. |
Interesting because #6138 (comment) represents better 1/3. |
Similarly, for the double case, the underlying representation has 52 fraction bits. The error for a number than cannot be represented exactly will be smaller, but still nonzero. In single precision, the value will be accurate to between 6 and 9 decimal digits, and double precision to between 15 and 17 decimal digits. All of this is explained in the following Wikipedia articles: https://en.wikipedia.org/wiki/Single-precision_floating-point_format and https://en.wikipedia.org/wiki/Double-precision_floating-point_format |
The "better representation" that you mention is the double precision case where there are more fractional bits available to encode the number. |
But please note: the accuracy of 6-9 decimal digits (single precision i.e. float) and 15-17 digits (double precision i.e. double) is a different thing than the exponent. In the situation that we are worried about, which is the conversion of a floating point number to a String that is NOT in exponential notation, we must worry about both. Let's consider two different situations: |
A properly-designed library will give results that are as accurate as possible within the constraints of the underlying number system. This has nothing to do with C vs FORTRAN because the underlying number system - IEEE 754 floating point - is the same in both cases. Sprintf, given a suitably large buffer and the appropriate arguments, does the right thing - it converts the underlying IEEE number to the corresponding string representation. That is the only right answer. Unfortunately, the corresponding string representation is long in some cases, so long that it could strain the resources of an 8-bit micro. It is not too long for a 32-bit processor, especially not going forward. |
My point of view about the whole discussion is: There are 2 topics here: 1- Solve the issue with Stack Smashing and the reset. For this issue, at a first glance, this PR does the job. 2- Getting Arduino String Class to print a float or double with the maximum precision possible. No solution at this time. @MitchBradley, if you want so, you or any other contributor can submit a PR to achieve it. SumaryI consider this PR ready for being reviewed regarding the first item here listed. @me-no-dev @pedrominatel @P-R-O-C-H-Y @PilnyTomas @atanisoft please let me know your thoughts. |
Making float work correctly and with precision is a difficult task. I was
asked as an undergraduate to improve the routines on our IBM360, and it
took a lot of effort and consultation with my professor to to be able to
perform calculations without losing precision.
I do not think I have the stomach to fix the errors here, but perhaps
someone can undertake to problem.
David
…On Sun., Jan. 16, 2022, 10:09 Rodrigo Garcia, ***@***.***> wrote:
My point of view about the whole discussion is:
There are 2 topics here:
1- Solve the issue with Stack Smashing and the reset.
For this issue, at a first glance, this PR does the job.
2- Getting Arduino String Class to print a float or double with the
maximum precision possible.
No solution at this time.
@MitchBradley <https://github.com/MitchBradley>, if you want so, you or
any other contributor can submit a PR to achieve it.
Sumary
I consider this PR ready for being reviewed regarding the first item here
listed.
@me-no-dev <https://github.com/me-no-dev> @pedrominatel
<https://github.com/pedrominatel> @P-R-O-C-H-Y
<https://github.com/P-R-O-C-H-Y> @PilnyTomas
<https://github.com/PilnyTomas> @atanisoft <https://github.com/atanisoft>
please let me know your thoughts.
—
Reply to this email directly, view it on GitHub
<#6138 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEDQSVDKL4PLA4XZGA2X6LUWMCWFANCNFSM5L6WOKNQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
@MitchBradley |
Please post a complete PR for evaluation. |
This is the testing sketch and results that I tried with the latest commit: void setup() {
Serial.begin(115200);
Serial.printf("\nTesting potential issues with Float-String:\n\n");
Serial.flush();
delay(1000);
// some edge cases
String s = String(1e38f, 0);
Serial.println(s);
Serial.println(String(1e39f, 0));
// comon test cases
Serial.println(String(1e32f, 0));
Serial.println(String(-1.0f, 30));
Serial.println(String(0.0f, 31));
Serial.println(String(0.0f, 100));
Serial.println(String(-234223.4f, 32));
double f = 1.0 / 3.0;
Serial.println(String(f, 100));
char fstr[512];
double d = 0.333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333;
sprintf(fstr, "%70.70f", d);
Serial.println();
Serial.printf("\nWith sprintf => [%s] \nand below with String()\n", fstr);
Serial.println(String((double)0.333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333, 99));
Serial.println();
}
void loop() {
// put your main code here, to run repeatedly:
} Output:
|
I observe the following about your test cases:
I think that the malloc/free formulation, in conjunction with the dtostrf() change to allow more than 255 digits, does the best possible job of doing what the programmer asked for within the inherent characteristics of the IEEE number system. |
I have commited your suggestion for the fix. void setup() {
Serial.begin(115200);
Serial.printf("\nTesting potential issues with Float-String:\n\n");
Serial.flush();
delay(1000);
Serial.println("==== MitchBradley test cases:");
Serial.println(String(1e-38, 40)); // smallest normalized single
Serial.println(String(-3.4e38, 40)); // largest single with extra precision digits
Serial.println(String(1e-45, 45)); // smallest denormal single
Serial.println(String((double)4.94e-324, 327)); // smallest denormal double
Serial.println(String((double)-1e308, 0)); // Largest double
Serial.println("==== MitchBradley test cases:");
}
void loop() {
// put your main code here, to run repeatedly:
} This is the output:
It seems there is a failure with |
The problem with 4.94e-324 is caused by the fact that the precision argument is unsigned char so the largest possible value is 255, which is less than the requested precision of 327. To fix that, it is necessary to make 3 changes:
|
@MitchBradley Everything is committed and it is now working as expected.
|
After width is changed from signed char to signed int, I think this PR will be ready. The width problem might not show up in the case where dtostrf() is called from String(), but I think it could cause strange formatting for direct uses of dtostrf(). You need to be able to pass in width values that are larger than the prec value, and that is impossible if width is char while the prec value can be more than 127. |
Perfect. All done. I think it shall be merged this week. |
I do understand where this 312 comes from: For ESP32 it may not be an immediate problem as it does have a lot more memory compared to ESP8266, but I guess this will be added to the ESP8266 code too. The exponent in the double is in the 12 most significant bits (including the sign). |
Your suggestion is indeed possible, but I doubt that it would make much difference in practice. The excess memory will be freed immediately. A malloc/free of several hundred bytes is no more likely to cause fragmentation than one of a smaller size. If you are already so low on memory that it exhausts the heap, it is very likely that you will crash soon anyway from allocations for incoming packets. Regarding the performance, when you are converting a double, you are already engaged in a slow operation involving lots of software floating point computations, so a bit more heap activity is unlikely to be a huge factor. |
I guess that last one is indeed the most important reason not to optimize. |
As implemented for ESP32, but not (yet) for ESP8266. See: espressif/arduino-esp32#6138
As implemented for ESP32, but not (yet) for ESP8266. See: espressif/arduino-esp32#6138
Summary
The PR fixes an issue related to Stack Smashing when trying to execute this sketch:
The wrong output is:
Expected correct output is:
Impact
None.
Related links
Fixes #5873