# Character types and Unicode

> C++ has six character types and four literal prefixes, but only the library's code-unit view of text. Knowing what a char holds, which encoding a literal gets, and what counts as a character for the reader is what keeps Unicode code correct.

- **Source:** C++ Atlas (https://cppatlas.com/topics/character-types-and-unicode)
- **Track:** Library utilities
- **Estimated reading time:** 13 minutes
- **ISO C++ clauses:** [basic.fundamental] (https://eel.is/c++draft/basic.fundamental), [lex.charset] (https://eel.is/c++draft/lex.charset), [lex.phases] (https://eel.is/c++draft/lex.phases), [lex.ccon] (https://eel.is/c++draft/lex.ccon), [lex.string] (https://eel.is/c++draft/lex.string), [lex.universal.char] (https://eel.is/c++draft/lex.universal.char), [lex.string.uchar] (https://eel.is/c++draft/lex.string.uchar), [format.string.std] (https://eel.is/c++draft/format.string.std), [depr.locale.stdcvt] (https://eel.is/c++draft/depr.locale.stdcvt), [depr.conversions] (https://eel.is/c++draft/depr.conversions)
- **cppreference links:** [Fundamental types](https://en.cppreference.com/w/cpp/language/types), [Character literal](https://en.cppreference.com/w/cpp/language/character_literal), [String literal](https://en.cppreference.com/w/cpp/language/string_literal), [Escape sequences](https://en.cppreference.com/w/cpp/language/escape), [Character sets and encodings](https://en.cppreference.com/w/cpp/language/charset), [std::isalpha](https://en.cppreference.com/w/cpp/string/byte/isalpha), [std::codecvt_utf8](https://en.cppreference.com/w/cpp/locale/codecvt_utf8), [Format specification](https://en.cppreference.com/w/cpp/utility/format/spec), [std::text_encoding](https://en.cppreference.com/w/cpp/text/text_encoding)
- **Prerequisites:** fundamental-types-and-arithmetic-conversions, strings-and-string-view

## The one idea

A C++ character type is an integer type that holds one **code unit**, which is a piece of an encoded string, not necessarily a character that a person would point to. A `char` of a UTF-8 string is one byte. A `char16_t` of a UTF-16 string is half of a surrogate pair for most emoji. Everything in this topic follows from asking which of three things you are counting: code units, code points, or what a reader sees as one character, called a grapheme cluster.

> **Mental model.** `std::string`, `std::u8string`, `std::u16string` and `std::u32string` are arrays of code units. The standard library, apart from a few narrow facilities, never looks inside them. Anything that needs code points or grapheme clusters, such as normalization, case folding or segmentation, is your job or a third-party library's.

## The six character types

`char`, `signed char` and `unsigned char` are three distinct types, even though `char` has the same representation as one of the other two. Which one is implementation-defined, and portable code must not rely on it. In practice `char` is signed on x86 Linux and unsigned on ARM Linux, and a plain `char` holding a byte above 127 gives a negative number where it is signed.

```cpp
#include <type_traits>

// All three are distinct types in every implementation.
static_assert(!std::is_same_v<char, signed char>);
static_assert(!std::is_same_v<char, unsigned char>);
```

`wchar_t` is the wide character type. Its size and its encoding are implementation-defined: it is 32 bits with UTF-32 on most Unix systems and 16 bits with UTF-16 on Windows. For that reason `wchar_t` is a poor choice for portable text.

`char16_t` and `char32_t` came with C++11 and are meant for UTF-16 and UTF-32 code units. Each has the same size, signedness and alignment as `std::uint_least16_t` and `std::uint_least32_t`. `char8_t` arrived in C++20 for UTF-8 code units, with the same size, signedness and alignment as `unsigned char`. It is a separate type, and it does not get the permission that `char`, `unsigned char` and `std::byte` have to alias other objects.

## Literal prefixes and their types

Each prefix selects an encoding and a type. The prefixes are `u8`, `u`, `U` and `L`, plus no prefix for ordinary literals.

| Literal | Element type | Encoding |
| --- | --- | --- |
| `"abc"` | `char` | ordinary literal encoding, implementation-defined |
| `u8"abc"` | `char8_t` since C++20, `char` before | UTF-8 |
| `u"abc"` | `char16_t` | UTF-16 |
| `U"abc"` | `char32_t` | UTF-32 |
| `L"abc"` | `wchar_t` | wide literal encoding, implementation-defined |

The ordinary and wide literal encodings are what the compiler uses when it writes your string into the program. GCC and Clang default to UTF-8, and you can change it with options such as GCC's `-fexec-charset`. The execution environment may have a completely different idea of text at run time, which is why printing a "UTF-8" literal can still produce garbage on a terminal configured differently.

```cpp
#include <iterator>

// Counting code units, not characters (the final element is the terminator).
constexpr auto a = std::size(u8"é");        // 3: U+00E9 is two UTF-8 code units
constexpr auto b = std::size(u"\U0001F600");     // 3: surrogate pair, two UTF-16 units
constexpr auto c = std::size(U"\U0001F600");     // 2: one UTF-32 unit
static_assert(a == 3 && b == 3 && c == 2);
```

A character literal holds a single code unit. `u'\U0001F600'` is ill-formed in C++23, because the code point needs two UTF-16 code units, while `U'\U0001F600'` is fine.

### The `char8_t` break

C++20 changed the type of UTF-8 literals from `const char` arrays to `const char8_t` arrays. Code that worked in C++17 stops compiling.

```cpp
// const char* p = u8"abc";
// OK in C++17, ill-formed in C++20: const char8_t* does not convert to const char*

// std::cout << u8"abc";
// OK in C++17, ill-formed in C++20: the overload for const char8_t* is deleted
```

Fixes are to use `std::u8string` and `std::u8string_view`, to cast through `reinterpret_cast<const char*>` where you must pass the bytes to a `char` API, or to drop the `u8` prefix and rely on an ordinary literal encoding of UTF-8. GCC has `-fno-char8_t` as an escape hatch while you migrate. Note that `char8_t` has no stream output at all, so UTF-8 text in iostreams still needs a `char` view.

> **Standardese.** [lex.ccon] says a character literal with a `u`, `U` or `u8` prefix must be encodable as a single code unit of its type, and in C++23 a literal that is not is ill-formed. [lex.string] gives each string prefix its element type and encoding, which is why `u8"..."` changed type in C++20 without any change to its bytes.

## Source, execution, and escapes

Three encodings are in play when you compile: the encoding of the source file, the ordinary literal encoding, and the encoding of the program's locale at run time. Before C++23 the standard talked about a basic source character set and an implementation-defined mapping to the execution set. C++23 reworked this around the translation character set, which is Unicode, and the ordinary and wide literal encodings. It also requires implementations to support UTF-8 source files.

Escapes let you avoid the question entirely for the source side. A universal-character-name, `\u` followed by four hex digits or `\U` followed by eight, names a code point whatever the source encoding is. C++23 added more forms.

```cpp
const char8_t* a = u8"α";                     // alpha, as a universal-character-name
const char8_t* b = u8"\N{GREEK SMALL LETTER ALPHA}"; // C++23: named escape
const char8_t* c = u8"\u{3b1}";                    // C++23: delimited escape
const char*    d = "\x{41}\o{102}";                // C++23: delimited hex and octal, "AB"
```

The delimited and named forms are useful because they have no length limit and the name documents the intent. The distinction between `\x41` and `A` matters: the first writes a raw code unit value, while the second names a code point and is encoded according to the literal's encoding.

## `<cctype>` and the negative char

The functions in `<cctype>`, such as `std::isalpha` and `std::toupper`, take an `int`. The argument must be representable as an `unsigned char` or equal `EOF`. Otherwise the behaviour is undefined. On a platform where `char` is signed, a byte such as 0xE9 becomes a negative `int`, so passing a plain `char` straight in is undefined.

```cpp
#include <cctype>

bool is_letter(char c) {
  return std::isalpha(static_cast<unsigned char>(c));  // the required cast
}
```

Even with the cast, `<cctype>` works one byte at a time in the current C locale. It cannot classify a multibyte UTF-8 character, and `std::toupper` cannot turn one code unit into two, so it cannot map `ß` to `SS`. For Unicode-aware classification and case mapping you need a library such as ICU.

> **Pitfall.** The cast to `unsigned char` fixes the undefined behaviour, not the model. `<cctype>` still classifies single bytes, so a UTF-8 text with accented letters gets misclassified even though no rule is broken.

## Code units, code points, grapheme clusters

The same text can have different lengths depending on what you count. The letter é can be one code point, U+00E9, or two: `e` followed by the combining acute accent U+0301. A reader sees one character either way.

```cpp
#include <string>

std::u8string s1 = u8"é";       // size() == 2 code units, 1 code point, 1 grapheme cluster
std::u8string s2 = u8"é";      // size() == 3 code units, 2 code points, 1 grapheme cluster
```

`size()` on any `basic_string` counts code units. Indexing, `find`, `substr` and comparisons are all by code unit too, so they can cut a multi-unit character in half, and `s1 == s2` is false even though they display the same. Normalization, collation and segmentation are not in the standard library.

The one place the library does look at graphemes is `std::format`. Its width and precision are measured in estimated terminal columns, not code units. For a string in a Unicode encoding, implementations should treat each extended grapheme cluster as 2 columns when it starts with a wide or fullwidth code point and 1 otherwise. The method is unspecified in general, so portable code should not depend on exact padding.

```cpp
#include <format>

// With a UTF-8 ordinary literal encoding, in libstdc++ 16:
auto w = std::format("{:>4}", "日");   // two spaces then the character: width 2
auto g = std::format("{:>4}", "é");  // three spaces: one grapheme cluster, width 1
```

Precision truncates a string to a prefix of whole clusters whose estimated width fits, so with that estimate a precision of 1 applied to a width-2 character yields an empty string.

## Conversions and `std::text_encoding`

`<codecvt>` and `std::wstring_convert` were the standard's UTF converters. Both were deprecated in C++17, and cppreference lists them as removed in C++26. Libstdc++ 16 still ships them, with deprecation warnings. The replacement in the standard is thinner: the C-style functions `mbrtoc8`, `mbrtoc16`, `mbrtoc32` and their inverses in `<cuchar>`, which convert between the locale's multibyte encoding and Unicode one code point at a time. For bulk conversion, write the loop yourself or use a library.

C++26 adds `std::text_encoding` in `<text_encoding>`, which is a C++26 feature. It does not convert anything. It identifies an encoding by its IANA registry number and name. `std::text_encoding::literal()` reports the ordinary literal encoding, and `std::text_encoding::environment()` reports the encoding the program's environment currently uses. The point is to let a program check before it assumes UTF-8.

```cpp
#include <text_encoding>

// C++26 only
static_assert(std::text_encoding::literal() == std::text_encoding::UTF8);
// Fails to compile if built with, for example, -fexec-charset=ISO-8859-1.
```

## What to remember

- A character type holds one code unit; text is a sequence of them in some encoding.
- `char`, `signed char` and `unsigned char` are three distinct types, and the signedness of `char` is implementation-defined.
- `wchar_t` has no portable encoding; `char8_t`, `char16_t` and `char32_t` are for UTF-8, UTF-16 and UTF-32.
- Since C++20, `u8` literals are `char8_t` arrays, which breaks conversion to `const char*` and stream output.
- A character literal must be encodable in one code unit; `u'\U0001F600'` is ill-formed.
- The ordinary literal encoding is chosen by the compiler, not by your source file; C++23 requires UTF-8 source support and adds named and delimited escapes.
- Pass `unsigned char` or `EOF` to `<cctype>` functions; a negative `char` is undefined.
- `size()` counts code units, so one displayed character can be one, two or more units, and canonically equal strings can compare unequal.
- `std::format` estimates width in columns, which is why precision can truncate to nothing.
- `<codecvt>` is deprecated and removed in C++26; `std::text_encoding` identifies encodings and is a C++26 feature.

## Concept Map Outline

- **Character types and Unicode:** A character type holds one code unit; what a reader sees as a character may span several.
  - **Character types:** Six types hold code units; only some have a portable encoding.
    - **char, signed, unsigned:** Three distinct types; the signedness of plain char is implementation-defined.
    - **wchar_t:** Size and encoding are implementation-defined, 16 bits on Windows and usually 32 elsewhere.
    - **char8_t, char16_t, char32_t:** Distinct types for UTF-8, UTF-16 and UTF-32 code units; char8_t arrived in C++20.
  - **Literals and encodings:** A prefix chooses the element type and the encoding of a literal.
    - **Prefixes u8, u, U, L:** u8 is UTF-8, u is UTF-16, U is UTF-32, and L uses the wide literal encoding.
    - **Ordinary literal encoding:** Chosen by the compiler, for example GCC's -fexec-charset, independent of the source file.
    - **Source encoding, C++23:** Implementations must support UTF-8 source files; the translation character set is Unicode.
    - **Char literal: one unit:** u'\U0001F600' is ill-formed in C++23 because the code point needs two UTF-16 units.
  - **Escapes:** Escapes name code points or raw code units without depending on the source encoding.
    - **Universal character names:** \u and \U name a code point and are encoded per the literal's encoding.
    - **Named and delimited, C++23:** \N{NAME}, \u{...}, \x{...} and \o{...} have no digit-count limit.
    - **\x versus \u:** \x writes a raw code unit value; \u names a code point.
  - **The char8_t break:** In C++20, u8 literals changed type and old code stopped compiling.
    - **Pointer conversion:** const char8_t* does not convert to const char*, so assigning a u8 literal to const char* fails.
    - **Stream output:** operator<< for char8_t and const char8_t* on narrow streams is deleted.
    - **Migration:** Use u8string, cast deliberately, or use an ordinary literal when it is UTF-8.
  - **Library limits:** The library counts code units and classifies single bytes; Unicode logic is yours.
    - **cctype negative char:** Arguments must fit unsigned char or equal EOF, so cast a plain char before calling.
    - **Code units versus graphemes:** size() counts code units; e plus a combining accent is three UTF-8 units but one grapheme.
    - **format width:** std::format estimates column width and truncates precision by whole grapheme clusters.
    - **No normalization:** The standard library has no normalization, collation or grapheme segmentation.
  - **Conversion and detection:** The old converters are on their way out; C++26 adds a way to identify encodings.
    - **codecvt deprecated:** Deprecated in C++17 and listed for removal in C++26; libstdc++ still ships it.
    - **cuchar functions:** mbrtoc8, mbrtoc16 and mbrtoc32 convert between the locale encoding and Unicode one code point at a time.
    - **text_encoding (C++26):** Identifies an encoding by IANA name and id; literal() reports the ordinary literal encoding.

## Standardese Quiz Questions

### Question 1 [warmup] ([lex.string], [basic.fundamental])

In C++20, what is the status of this program?

```cpp
const char* p = u8"abc";

int main() { return 0; }
```

A. Well-formed: a UTF-8 literal is an array of `const char`
B. Ill-formed: `const char8_t*` does not convert to `const char*`
C. Well-formed, but the literal is converted to the ordinary literal encoding
D. Undefined behavior at the conversion

**Answer (B):** Ill-formed: `const char8_t*` does not convert to `const char*`

**Why:** Since C++20 a `u8` string literal is an array of `const char8_t`, and `char8_t` is a distinct type, so there is no implicit conversion to `const char*`. In C++17 the literal was an array of `const char`, which is why this code used to compile and why the first option feels right. The diagnosis is a compile-time error, not a run-time problem.

### Question 2 [warmup] ([lex.string], [lex.charset])

In C++17 or later, what is printed? The arrays include the terminating null, which the code subtracts.

```cpp
#include <cstdio>
#include <iterator>

int main() {
  std::printf("%zu %zu %zu\n", std::size(u8"\u00e9") - 1,
              std::size(u"\U0001F600") - 1, std::size(U"\U0001F600") - 1);
}
```

A. 2 2 1
B. 2 1 1
C. 1 1 1
D. 2 2 2

**Answer (A):** 2 2 1

**Why:** `u8"\u00e9"` is encoded as UTF-8, where U+00E9 takes two code units. `u"\U0001F600"` is UTF-16, where a code point above U+FFFF becomes a surrogate pair of two code units, and `U"\U0001F600"` is UTF-32, where it is one unit. The option `1 1 1` counts code points, not the code units the arrays hold.

### Question 3 [warmup] ([lex.string])

In C++20, what is the status of this program?

```cpp
#include <iostream>

int main() {
  std::cout << u8"abc";
}
```

A. Well-formed: it prints `abc`
B. Well-formed: it prints a pointer value
C. Ill-formed: the stream insertion for `const char8_t*` is deleted
D. Undefined behavior: the stream reads past the array

**Answer (C):** Ill-formed: the stream insertion for `const char8_t*` is deleted

**Why:** C++20 added deleted overloads of `operator<<` for `char8_t` and `const char8_t*` on narrow streams, specifically to stop `u8` literals from printing as a pointer or as numbers. Overload resolution selects the deleted function, which makes the program ill-formed. In C++17 the same line compiled and printed `abc`, because the literal was `const char`.

### Question 4 [tricky] ([lex.string], [basic.fundamental])

In C++20 or later, what is printed?

```cpp
#include <cstdio>
#include <string>

int main() {
  std::u8string s = u8"e\u0301";
  std::printf("%zu\n", s.size());
}
```

A. 1
B. 2
C. 4
D. 3

**Answer (D):** 3

**Why:** `size()` counts code units. The letter `e` is one UTF-8 code unit and U+0301, the combining acute accent, is two, so the total is 3. It is also two code points and one grapheme cluster, which is why neither `1` nor `2` is a code-unit count.

### Question 5 [tricky] ([basic.fundamental])

What is printed on every conforming implementation?

```cpp
#include <cstdio>
#include <type_traits>

int main() {
  std::printf("%d\n", std::is_same_v<char, signed char> +
                       std::is_same_v<char, unsigned char>);
}
```

A. 1
B. 2
C. 0
D. It depends on whether `char` is signed

**Answer (C):** 0

**Why:** `char`, `signed char` and `unsigned char` are three distinct types in every implementation. Plain `char` behaves like one of the other two, and which one is implementation-defined, but it is never the same type as either. The tempting answer `1` confuses the representation with the type, and the last option is wrong because the result does not change with signedness.

### Question 6 [tricky] ([lex.ccon])

In C++23, what is the status of this declaration?

```cpp
char16_t c = u'\U0001F600';

int main() { return 0; }
```

A. Well-formed: it holds the high surrogate
B. Ill-formed: the code point is not encodable in a single `char16_t`
C. Well-formed: the value is truncated to 16 bits
D. Implementation-defined value

**Answer (B):** Ill-formed: the code point is not encodable in a single `char16_t`

**Why:** A character literal holds a single code unit. U+1F600 needs two UTF-16 code units, so it cannot be represented by a `char16_t` character literal, and C++23 makes such a literal ill-formed. Both compilers reject it, and neither truncates silently. `U'\U0001F600'` would be fine, since `char32_t` holds the whole code point.

### Question 7 [standardese] ([basic.fundamental])

Assume `char` is signed, `'\xE9'` converts to -23, and `EOF` is -1. Which function has undefined behavior when called as shown?

```cpp
#include <cctype>
#include <cstdio>

bool a(char c) { return std::isalpha(c); }
bool b(char c) { return std::isalpha(static_cast<unsigned char>(c)); }
bool c() { return std::isalpha(EOF); }

int main() { return b('\xE9') + c(); }
```

A. `a`, because `-23` is not representable as `unsigned char` and is not `EOF`
B. `b`, because the cast to `unsigned char` can overflow
C. `c`, because `EOF` is not a valid argument
D. None: `std::isalpha` accepts any `int`

**Answer (A):** `a`, because `-23` is not representable as `unsigned char` and is not `EOF`

**Why:** The `<cctype>` functions require an argument representable as `unsigned char` or equal to `EOF`; anything else is undefined. In `a`, a plain `char` with a negative value is passed unchanged, which breaks that rule. In `b` the cast yields 233, which is valid, and `c` passes `EOF`, which is explicitly allowed. The last option mistakes the `int` parameter type for the actual domain.

### Question 8 [standardese] ([depr.locale.stdcvt], [depr.conversions])

A C++23 program uses `std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t>` to convert UTF-32 to UTF-8. What does the C++23 standard say?

```cpp
#include <codecvt>
#include <locale>
#include <string>

int main() {
  std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t> conv;
  std::string s = conv.to_bytes(U"abc");
  return s.size() == 3 ? 0 : 1;
}
```

A. Ill-formed: both were removed in C++20
B. The program is valid, and this is the recommended way to convert UTF encodings
C. The behavior is undefined because `codecvt_utf8` only supports UCS-2
D. The program is valid, and both facilities are deprecated, so a compiler may warn

**Answer (D):** The program is valid, and both facilities are deprecated, so a compiler may warn

**Why:** `<codecvt>` and `std::wstring_convert` were deprecated in C++17 and remain in the C++23 library, so using them is valid and typically produces a deprecation warning. They are listed for removal in C++26. The deprecation is the reason not to call them recommended, and the `char32_t` instantiation is supported, so it is not undefined.
