Qbasicnews.com
March 29, 2020, 08:22:03 AM
 Pages: [1]
 Author Topic: 1.#INF, how I hate thee  (Read 1980 times)
pr0gger
I hold this place together

Posts: 775

 « on: August 21, 2004, 03:57:35 PM »

I'm running into some problems with single precision variables in my program.  Every once in a while, the program will generate an error (Overflow), and I'll take a look at some of the variables and find that the problem arises whenever one (or more) of these values appear: when an operation is supposed to return zero... but one of these appears instead  (That's what's so frustrating-- it's not my fault, it's a result of binary math).

Is there any way to fix this?
 Logged

size=9]"To announce that there must be no criticism of the president, or that we are to stand by the president, right or wrong, is not only unpatriotic and servile, but is morally treasonable to the American public." -- Theodore Roosevelt[/size]
Nexinarus
Wandering Guru

Posts: 301

 « Reply #1 on: August 21, 2004, 05:38:07 PM »

hmm can you show some simple code of the problem you mean?
 Logged
Moneo
Na_th_an

Posts: 1971

 « Reply #2 on: August 21, 2004, 09:45:04 PM »

Progger,

Do you have to use single precision floating point?
Let's asume you need precision to 2 decimal places, then:
* Use LONG variables.
* Use 3 IMPLIED decimal places, so at the end you can round to your required 2 decimal places.
* If the input to your program has 2 decimal places, multiply these numbers by 1000 to convert them to 3 implied decimal places while actually using a whole number as LONG.
* Do your calculations using the LONG numbers.
* At the end, divide you answer(s) by 100 while rounding, storing it into a floating point number, which will give you an answer with 2 decimal places.

I do this all the time to avoid all the crazy problems you get with floating point numbers.
*****
 Logged
pr0gger
I hold this place together

Posts: 775

 « Reply #3 on: August 23, 2004, 11:45:03 AM »

Nex: I can't, actually... it's a very large program.
Moneo: I'm using some trig equations, so I try to be as precise as possible (i.e., 3-4 decimal places).    someone once told me that QB division is pretty slow, but this is really getting on my nerves so I'm gonna try that.

thanks
 Logged

size=9]"To announce that there must be no criticism of the president, or that we are to stand by the president, right or wrong, is not only unpatriotic and servile, but is morally treasonable to the American public." -- Theodore Roosevelt[/size]
 Pages: [1]