Found with m-c 20190909-6f423e980a92

Built with undefined behavior sanitizer checks enabled via mozconfig.
ac_add_options --enable-undefined-sanitizer="shift"

src/gfx/cairo/libpixman/src/pixman.c:345:10: runtime error: left shift of negative value -1
    #0 0x7fbfcf8661e3 in compute_transformed_extents src/gfx/cairo/libpixman/src/pixman.c
    #1 0x7fbfcf863eb8 in analyze_extent src/gfx/cairo/libpixman/src/pixman.c:536:10
    #2 0x7fbfcf862fec in _moz_pixman_image_composite32 src/gfx/cairo/libpixman/src/pixman.c:643:10
    #3 0x7fbfcf63d0f6 in _composite_boxes src/gfx/cairo/cairo/src/cairo-image-surface.c:3051:3
    #4 0x7fbfcf63d0f6 in _clip_and_composite_boxes src/gfx/cairo/cairo/src/cairo-image-surface.c:3090
    #5 0x7fbfcf62d219 in _cairo_image_surface_paint src/gfx/cairo/cairo/src/cairo-image-surface.c:3338:11
    #6 0x7fbfcf69a43a in _cairo_surface_paint src/gfx/cairo/cairo/src/cairo-surface.c:2110:11
    #7 0x7fbfcf61e779 in _cairo_gstate_paint src/gfx/cairo/cairo/src/cairo-gstate.c:1049:14
    #8 0x7fbfcf6bf100 in _moz_cairo_paint src/gfx/cairo/cairo/src/cairo.c:2252:14
    #9 0x7fbfcf6bf3a1 in _moz_cairo_paint_with_alpha src/gfx/cairo/cairo/src/cairo.c:2280:2
    #10 0x7fbfc7dc324b in mozilla::gfx::DrawTargetCairo::DrawSurface(mozilla::gfx::SourceSurface*, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::DrawSurfaceOptions const&, mozilla::gfx::DrawOptions const&) src/gfx/2d/DrawTargetCairo.cpp:828:3
    #11 0x7fbfc7e58f87 in mozilla::gfx::FilterNodeTransformSoftware::Render(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:1206:7
    #12 0x7fbfc7e50dd6 in mozilla::gfx::FilterNodeSoftware::GetOutput(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:638:20
    #13 0x7fbfc7e54755 in mozilla::gfx::FilterNodeSoftware::GetInputDataSourceSurface(unsigned int, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::FilterNodeSoftware::FormatHint, mozilla::gfx::ConvolveMatrixEdgeMode, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) src/gfx/2d/FilterNodeSoftware.cpp:770:25
    #14 0x7fbfc7e6d740 in mozilla::gfx::FilterNodeUnpremultiplySoftware::Render(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:3227:7
    #15 0x7fbfc7e50dd6 in mozilla::gfx::FilterNodeSoftware::GetOutput(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:638:20
    #16 0x7fbfc7e54755 in mozilla::gfx::FilterNodeSoftware::GetInputDataSourceSurface(unsigned int, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::FilterNodeSoftware::FormatHint, mozilla::gfx::ConvolveMatrixEdgeMode, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) src/gfx/2d/FilterNodeSoftware.cpp:770:25
    #17 0x7fbfc7e6d01b in mozilla::gfx::FilterNodeCropSoftware::Render(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:3166:10
    #18 0x7fbfc7e50dd6 in mozilla::gfx::FilterNodeSoftware::GetOutput(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:638:20
    #19 0x7fbfc7e54755 in mozilla::gfx::FilterNodeSoftware::GetInputDataSourceSurface(unsigned int, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::FilterNodeSoftware::FormatHint, mozilla::gfx::ConvolveMatrixEdgeMode, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) src/gfx/2d/FilterNodeSoftware.cpp:770:25
    #20 0x7fbfc7e6d540 in mozilla::gfx::FilterNodePremultiplySoftware::Render(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:3196:7
    #21 0x7fbfc7e50dd6 in mozilla::gfx::FilterNodeSoftware::GetOutput(mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&) src/gfx/2d/FilterNodeSoftware.cpp:638:20
    #22 0x7fbfc7e500b2 in mozilla::gfx::FilterNodeSoftware::Draw(mozilla::gfx::DrawTarget*, mozilla::gfx::RectTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::PointTyped<mozilla::gfx::UnknownUnits, float> const&, mozilla::gfx::DrawOptions const&) src/gfx/2d/FilterNodeSoftware.cpp:572:14
    #23 0x7fbfc7e20551 in mozilla::gfx::DrawFilterCommand::ExecuteOnDT(mozilla::gfx::DrawTarget*, mozilla::gfx::BaseMatrix<float> const*) const src/gfx/2d/DrawCommands.h:223:10
    #24 0x7fbfc7dbe6d5 in mozilla::gfx::DrawTargetCaptureImpl::ReplayToDrawTarget(mozilla::gfx::DrawTarget*, mozilla::gfx::BaseMatrix<float> const&) src/gfx/2d/DrawTargetCapture.cpp:330:10
    #25 0x7fbfc7dbe523 in mozilla::gfx::DrawTarget::DrawCapturedDT(mozilla::gfx::DrawTargetCapture*, mozilla::gfx::BaseMatrix<float> const&) src/gfx/2d/DrawTarget.cpp:168:9
    #26 0x7fbfc81807c6 in mozilla::layers::PaintThread::AsyncPaintTask(mozilla::layers::CompositorBridgeChild*, mozilla::layers::PaintTask*) src/gfx/layers/PaintThread.cpp:206:13
    #27 0x7fbfc81c3381 in operator() src/gfx/layers/PaintThread.cpp:178:38
    #28 0x7fbfc81c3381 in mozilla::detail::RunnableFunction<mozilla::layers::PaintThread::QueuePaintTask(mozilla::UniquePtr<mozilla::layers::PaintTask, mozilla::DefaultDelete<mozilla::layers::PaintTask> >&&)::$_7>::Run() src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:564
    #29 0x7fbfc50276bf in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1225:14
    #30 0x7fbfc502d59d in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:486:10
    #31 0x7fbfc63e9d3c in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:333:5
    #32 0x7fbfc627d607 in RunInternal src/ipc/chromium/src/base/message_loop.cc:315:10
    #33 0x7fbfc627d607 in RunHandler src/ipc/chromium/src/base/message_loop.cc:308
    #34 0x7fbfc627d607 in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:290
    #35 0x7fbfc5020ed0 in nsThread::ThreadFunc(void*) src/xpcom/threads/nsThread.cpp:458:11
    #36 0x7fbfe819efde in _pt_root src/nsprpub/pr/src/pthreads/ptthread.c:198:5
    #37 0x7fbfe7df06da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

The issue here seems to be the pixman_int_to_fixed macro, which left-shifts its argument; this is undefined behavior if the argument is negative -- this would even include the trivial case of the pixman_fixed_minus_1 macro a couple of lines earlier.

I think the simplest solution is to cast the argument to uint32_t before shifting. If I'm reading the standard correctly, we'll still be in the realm of implementation-defined behavior at the point where the shifted result is cast back to (signed) pixman_fixed_t, but that's a better place to be than undefined behavior.

Also reported upstream in https://gitlab.freedesktop.org/pixman/pixman/merge_requests/16.

https://hg.mozilla.org/mozilla-central/rev/3930b95096f5