This was found by enabling the float-cast-overflow check in UBSan and running existing tests. This type of issue can create inconsistencies across platforms, architectures and optimization levels.

This issue is found in the existing test: toolkit/components/normandy/test/unit/test_addon_unenroll.js

To enable this check add the following to your mozconfig:

ac_add_options --enable-undefined-sanitizer="float-cast-overflow"

WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/normandy/test/unit/test_addon_unenroll.js | xpcshell return code: 1
INFO -  PID 1283 | /builds/worker/checkouts/gecko/xpcom/ds/nsVariant.cpp:518:1: runtime error: nan is outside the range of representable values of type 'unsigned int'
INFO -  PID 1283 |     #0 0x7f4b624644f7 in nsDiscriminatedUnion::ConvertToUint32(unsigned int*) const /builds/worker/checkouts/gecko/xpcom/ds/nsVariant.cpp:518:1
INFO -  PID 1283 |     #1 0x7f4b6d8eeebb in (anonymous namespace)::ScalarUnsigned::SetValue(nsIVariant*) /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/TelemetryScalar.cpp:514:7
INFO -  PID 1283 |     #2 0x7f4b6d8dc33e in SetValue /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/TelemetryScalar.cpp:921:18
INFO -  PID 1283 |     #3 0x7f4b6d8dc33e in (anonymous namespace)::internal_UpdateKeyedScalar(mozilla::detail::BaseAutoLock<mozilla::StaticMutex&> const&, nsTSubstring<char> const&, nsTSubstring<char16_t> const&, mozilla::Telemetry::ScalarActionType, nsIVariant*, mozilla::Telemetry::ProcessID, bool) /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/TelemetryScalar.cpp:1899:20
INFO -  PID 1283 |     #4 0x7f4b6d8df6b6 in TelemetryScalar::Set(nsTSubstring<char> const&, nsTSubstring<char16_t> const&, JS::Handle<JS::Value>, JSContext*) /builds/worker/checkouts/gecko/toolkit/components/telemetry/core/TelemetryScalar.cpp:2726:10
INFO -  PID 1283 |     #5 0x7f4b625ec675 in NS_InvokeByIndex /builds/worker/checkouts/gecko/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:101
INFO -  PID 1283 |     #6 0x7f4b641c3ebe in Invoke /builds/worker/checkouts/gecko/js/xpconnect/src/XPCWrappedNative.cpp:1631:10
INFO -  PID 1283 |     #7 0x7f4b641c3ebe in Call /builds/worker/checkouts/gecko/js/xpconnect/src/XPCWrappedNative.cpp:1184:19
INFO -  PID 1283 |     #8 0x7f4b641c3ebe in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) /builds/worker/checkouts/gecko/js/xpconnect/src/XPCWrappedNative.cpp:1130:23
INFO -  PID 1283 |     #9 0x7f4b641c88c4 in XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) /builds/worker/checkouts/gecko/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:923:10
INFO -  PID 1283 |     #10 0x7f4b6dcc6ce4 in CallJSNative /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:425:13
INFO -  PID 1283 |     #11 0x7f4b6dcc6ce4 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:512:12
INFO -  PID 1283 |     #12 0x7f4b6dcb3159 in CallFromStack /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:576:10
INFO -  PID 1283 |     #13 0x7f4b6dcb3159 in Interpret(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:3309:16
INFO -  PID 1283 |     #14 0x7f4b6dc97ed1 in js::RunScript(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:394:13
INFO -  PID 1283 |     #15 0x7f4b6dcc6e1f in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:544:13
INFO -  PID 1283 |     #16 0x7f4b6dcc8f6b in js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:589:8
INFO -  PID 1283 |     #17 0x7f4b6e1e7ed7 in js::CallSelfHostedFunction(JSContext*, JS::Handle<js::PropertyName*>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>) /builds/worker/checkouts/gecko/js/src/vm/SelfHosting.cpp:1539:10
INFO -  PID 1283 |     #18 0x7f4b6ea8c42a in js::jit::InterpretResume(JSContext*, JS::Handle<JSObject*>, JS::Value*, JS::MutableHandle<JS::Value>) /builds/worker/checkouts/gecko/js/src/jit/VMFunctions.cpp:1093:10

It looks like CASE__NUMERIC_CONVERSION_DOUBLE_MIN_MAX_INT doesn't account for nan. Maybe it needs to explicitly check for it and return an error then?

Doing a direct cast produces an error in UBSan.

This is probably less bad than what it is doing. I think the only
difference is that for doubles we won't just do a straight cast,
but will instead do some bounds checking. Thanks to the prior patch,
it will also produce an error for NaN. This might make us start to
fail if you try to convert, say, 3.3 to a char but hopefully that
doesn't matter.

The existing code treats all non-0.0 values as true, but in
Javascript NaN is also falsy, so let's just do that here.

Also add some braces around multi-line ifs.

I added a patch to fix the pre-existing issue with macro arguments not having parens around them.

Backed out for causing bustage on TestVariant.cpp

• backout: https://hg.mozilla.org/integration/autoland/rev/0ea33ecdf904896bbd1b4e39b35f9ae0a79375ef
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=6a33352c82991ad7da369a72f0bc898c8bfb6927
• push where bustages have shown up: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=2abaeed5247f15ea79a4be205551bae3587f9507&searchStr=build&selectedTaskRun=ZcsGppJIQLyDjy4uCSEnWQ.0
• failure log:
  • ERROR - /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp:39:15: error: use of undeclared identifier 'isnan'
  • ERROR - /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp:9:6: error: use of undeclared identifier 'Variant'; did you mean 'mozilla::Variant'?

[task 2022-02-02T23:15:34.589Z] 23:15:34     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/xpcom/tests/gtest'
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -std=gnu++17 -o TestVariant.o -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -I/builds/worker/workspace/obj-build/dist/system_wrappers -include /builds/worker/checkouts/gecko/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/xpcom/tests/gtest -I/builds/worker/workspace/obj-build/xpcom/tests/gtest -I/builds/worker/checkouts/gecko/xpcom/base -I/builds/worker/checkouts/gecko/toolkit/components/telemetry/tests/gtest -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -DMOZILLA_CLIENT -include /builds/worker/workspace/obj-build/mozilla-config.h -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wcomma -Wenum-compare-conditional -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=free-nonheap-object -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-psabi -Wno-unknown-warning-option -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fcrash-diagnostics-dir=/builds/worker/artifacts -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -funwind-tables -Werror -fno-strict-aliasing  -MD -MP -MF .deps/TestVariant.o.pp   /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp
[task 2022-02-02T23:15:34.599Z] 23:15:34    ERROR -  /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp:9:6: error: use of undeclared identifier 'Variant'; did you mean 'mozilla::Variant'?
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  TEST(Variant, DoubleNaN)
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -       ^~~~~~~
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -       mozilla::Variant
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  /builds/worker/workspace/obj-build/dist/include/mozilla/Variant.h:570:69: note: 'mozilla::Variant' declared here
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  class MOZ_INHERIT_TYPE_ANNOTATIONS_FROM_TEMPLATE_ARGS MOZ_NON_PARAM Variant {
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -                                                                      ^
[task 2022-02-02T23:15:34.599Z] 23:15:34    ERROR -  /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp:9:6: error: unknown type name 'Variant'; did you mean 'nsVariant'?
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  TEST(Variant, DoubleNaN)
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -       ^~~~~~~
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -       nsVariant
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  /builds/worker/workspace/obj-build/dist/include/nsVariant.h:190:7: note: 'nsVariant' declared here
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  class nsVariant final : public nsVariantBase {
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -        ^
[task 2022-02-02T23:15:34.599Z] 23:15:34    ERROR -  /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestVariant.cpp:9:15: error: unknown type name 'DoubleNaN'
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -  TEST(Variant, DoubleNaN)
[task 2022-02-02T23:15:34.599Z] 23:15:34     INFO -                ^

Strange that that doesn't show up for me on OSX when I build locally. I guess I'm missing some includes and it is picking them up via the unified build.

https://hg.mozilla.org/mozilla-central/rev/539ab431b41c
https://hg.mozilla.org/mozilla-central/rev/a0dafeaa3bdf
https://hg.mozilla.org/mozilla-central/rev/fd11a19b7741
https://hg.mozilla.org/mozilla-central/rev/ff8ec2539fd0
https://hg.mozilla.org/mozilla-central/rev/4ec74b34140e
https://hg.mozilla.org/mozilla-central/rev/155bf6555b4e