From 42ce199c9cfe129e5e21afd48dfe757a6acf87c4 Mon Sep 17 00:00:00 2001
From: DRC <information@libjpeg-turbo.org>
Date: Tue, 4 Apr 2023 19:06:20 -0500
Subject: [PATCH] Decomp: Don't enable 2-pass color quant w/ RGB565

The 2-pass color quantization algorithm assumes 3-sample pixels.  RGB565
is the only 3-component colorspace that doesn't have 3-sample pixels, so
we need to treat it as a special case when determining whether to enable
2-pass color quantization.  Otherwise, attempting to initialize 2-pass
color quantization with an RGB565 output buffer could cause
prescan_quantize() to read from uninitialized memory and subsequently
underflow/overflow the histogram array.

djpeg is supposed to fail gracefully if both -rgb565 and -colors are
specified, because none of its destination managers (image writers)
support color quantization with RGB565.  However, prescan_quantize() was
called before that could occur.  It is possible but very unlikely that
these issues could have been reproduced in applications other than
djpeg.  The issues involve the use of two features (12-bit precision and
RGB565) that are incompatible, and they also involve the use of two
rarely-used legacy features (RGB565 and color quantization) that don't
make much sense when combined.

Fixes #668
Fixes #671
Fixes #680

CVE: CVE-2023-2804
Upstream-Status: Backport [https://github.com/libjpeg-turbo/libjpeg-turbo/commit/42ce199c9cfe129e5e21afd48dfe757a6acf87c4]

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 ChangeLog.md | 12 ++++++++++++
 jdmaster.c   |  5 +++--
 jquant2.c    |  5 +++--
 3 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/ChangeLog.md b/ChangeLog.md
index 1c1e6538a..f1bfb3d87 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -1,3 +1,15 @@
+2.1.6
+=====
+
+### Significant changes relative to 2.1.5.1:
+
+1. Fixed an oversight in 1.4 beta1[8] that caused various segfaults and buffer
+overruns when attempting to decompress various specially-crafted malformed
+12-bit-per-component JPEG images using a 12-bit-per-component build of djpeg
+(`-DWITH_12BIT=1`) with both color quantization and RGB565 color conversion
+enabled.
+
+
 2.1.5.1
 =======
 
diff --git a/jdmaster.c b/jdmaster.c
index a3690bf56..a9446adfd 100644
--- a/jdmaster.c
+++ b/jdmaster.c
@@ -5,7 +5,7 @@
  * Copyright (C) 1991-1997, Thomas G. Lane.
  * Modified 2002-2009 by Guido Vollbeding.
  * libjpeg-turbo Modifications:
- * Copyright (C) 2009-2011, 2016, 2019, 2022, D. R. Commander.
+ * Copyright (C) 2009-2011, 2016, 2019, 2022-2023, D. R. Commander.
  * Copyright (C) 2013, Linaro Limited.
  * Copyright (C) 2015, Google, Inc.
  * For conditions of distribution and use, see the accompanying README.ijg
@@ -480,7 +480,8 @@ master_selection(j_decompress_ptr cinfo)
     if (cinfo->raw_data_out)
       ERREXIT(cinfo, JERR_NOTIMPL);
     /* 2-pass quantizer only works in 3-component color space. */
-    if (cinfo->out_color_components != 3) {
+    if (cinfo->out_color_components != 3 ||
+        cinfo->out_color_space == JCS_RGB565) {
       cinfo->enable_1pass_quant = TRUE;
       cinfo->enable_external_quant = FALSE;
       cinfo->enable_2pass_quant = FALSE;
diff --git a/jquant2.c b/jquant2.c
index 44efb18ca..1c14ef763 100644
--- a/jquant2.c
+++ b/jquant2.c
@@ -4,7 +4,7 @@
  * This file was part of the Independent JPEG Group's software:
  * Copyright (C) 1991-1996, Thomas G. Lane.
  * libjpeg-turbo Modifications:
- * Copyright (C) 2009, 2014-2015, 2020, D. R. Commander.
+ * Copyright (C) 2009, 2014-2015, 2020, 2023, D. R. Commander.
  * For conditions of distribution and use, see the accompanying README.ijg
  * file.
  *
@@ -1230,7 +1230,8 @@ jinit_2pass_quantizer(j_decompress_ptr cinfo)
   cquantize->error_limiter = NULL;
 
   /* Make sure jdmaster didn't give me a case I can't handle */
-  if (cinfo->out_color_components != 3)
+  if (cinfo->out_color_components != 3 ||
+      cinfo->out_color_space == JCS_RGB565)
     ERREXIT(cinfo, JERR_NOTIMPL);
 
   /* Allocate the histogram/inverse colormap storage */
