diff mbox series

[v2,01/17] iio: core: Increase precision of IIO_VAL_FRACTIONAL_LOG2 when possible

Message ID 20220619185839.1363503-2-jic23@kernel.org (mailing list archive)
State Changes Requested
Headers show
Series staging/iio: Clean up AD7746 CDC driver and move from staging. | expand

Commit Message

Jonathan Cameron June 19, 2022, 6:58 p.m. UTC 
From: Jonathan Cameron <Jonathan.Cameron@huawei.com>

With some high resolution sensors such as the ad7746 the
build up of error when multiplying the _raw and _scale
values together can be significant.  Reduce this affect by
providing additional resolution in both calculation and
formatting of result. If overflow would occur with a 1e12
multiplier, fall back to the 1e9 used before this patch.

Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 drivers/iio/industrialio-core.c | 30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)
diff mbox series

Patch

diff --git a/drivers/iio/industrialio-core.c b/drivers/iio/industrialio-core.c
index dc3e1cb9bfbd..6b401f63805f 100644
--- a/drivers/iio/industrialio-core.c
+++ b/drivers/iio/industrialio-core.c
@@ -674,14 +674,28 @@  static ssize_t __iio_format_value(char *buf, size_t offset, unsigned int type,
 		else
 			return sysfs_emit_at(buf, offset, "%d.%09u", tmp0,
 					     abs(tmp1));
-	case IIO_VAL_FRACTIONAL_LOG2:
-		tmp2 = shift_right((s64)vals[0] * 1000000000LL, vals[1]);
-		tmp0 = (int)div_s64_rem(tmp2, 1000000000LL, &tmp1);
-		if (tmp0 == 0 && tmp2 < 0)
-			return sysfs_emit_at(buf, offset, "-0.%09u", abs(tmp1));
-		else
-			return sysfs_emit_at(buf, offset, "%d.%09u", tmp0,
-					     abs(tmp1));
+	case IIO_VAL_FRACTIONAL_LOG2: {
+		u64 t1, t2, mult;
+		int integer, precision;
+		bool neg = vals[0] < 0;
+
+		if (vals[0] > ULLONG_MAX / PICO) {
+			mult = NANO;
+			precision = 9;
+		} else {
+			mult = PICO;
+			precision = 12;
+		}
+		t1 = shift_right((u64)abs(vals[0]) * mult, vals[1]);
+		integer = (int)div64_u64_rem(t1, mult, &t2);
+		if (integer == 0 && neg)
+			return sysfs_emit_at(buf, offset, "-0.%0*llu",
+					     precision, abs(t2));
+		if (neg)
+			integer *= -1;
+		return sysfs_emit_at(buf, offset, "%d.%0*llu", integer,
+				     precision, abs(t2));
+	}
 	case IIO_VAL_INT_MULTIPLE:
 	{
 		int i;