Monday, March 12, 2012

Problems with float results in calculation.

I have a stored procedure that uses the LOG10 function to calculate a logarithmic average. The first part of the calculation takes 1 divided by the number of results achieved per group. This always results in something like 0.2 or 0.3. Now, the problem is that SQL Server rounds off the result to the nearest 0, which always results in a result of 0. This then causes problems with the LOG10 function which results in a "Domain error" message. Anyone know of a way I can tell SQL to keep the value of 0.3 or 0.2 when it is calculated and not do a rounding? I've tried to convert the result of the calculation to float or real, but this did not help. Anyone know how to fix this?

Thanks,

Bcs you are doing integer division.

use the following sampel to fix yours..

Code Snippet

Select

1/Count(*) [1/Count(*)], --Wrong, Integer Division

1.0/Count(*) [1.0/Count(*)], --Partially Correct

1.0/Cast(Count(*) as Float) [1.0/Cast(Count(*) as Float)], --Perfect Expression

Log10(1.0/Cast(Count(*) as Float)) [Your Expression] --This is your answer

From

Sysobjects

|||Thanks a lot. It helped.

No comments:

Post a Comment