Character Model for the World Wide Web 1.0

3.1.1 Introduction

The glossary entry in [Unicode 3.0] gives:

"Character. (1) The smallest component of written language that has semantic values; refers to the abstract meaning and/or shape ..."

The word 'character' is used in many contexts, with different meanings. Human cultures have radically differing writing systems, leading to radically differing concepts of a character. Such wide variation in end user experience can, and often does, result in misunderstanding. This variation is sometimes mistakenly seen as the consequence of imperfect technology. Instead, it derives from the great flexibility and creativity of the human mind and the long tradition of writing as an important part of the human cultural heritage. The alphabetic approach used by scripts such as Latin, Cyrillic and Greek is only one of several possibilities.

The developers of W3C specifications, and the developers of software based on those specifications, are likely to be more familiar with usages they have experienced and less familiar with the wide variety of usages in an international context. Furthermore, within a computing context, characters are often confused with related concepts, resulting in incomplete or inappropriate specifications and software.

This section examines some of these contexts, meanings and confusions.

3.1.2 Units of aural rendering

In some scripts, characters have a close relationship to phonemes (a phoneme is a minimally distinct sound in the context of a particular spoken language), while in others they are closely related to meanings. Even when characters (loosely) correspond to phonemes, this relationship may not be simple, and there is rarely a one-to-one correspondence between character and phoneme.

[S] [I] Specifications and software MUST NOT assume that there is a one-to-one correspondence between characters and the sounds of a language.

3.1.3 Units of visual rendering

Visual rendering introduces the notion of a glyph. Glyphs are defined by ISO/IEC 9541-1 [ISO/IEC 9541-1] as "a recognizable abstract graphic symbol which is independent of a specific design". There is not a one-to-one correspondence between characters and glyphs:

• A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character). These glyphs may be physically separated from one another.

• A single glyph may represent a sequence of characters (this is the case with ligatures, among others).

• A character may be rendered with very different glyphs depending on the context.

• A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A).

Each glyph can be represented by a number of different glyph images; a set of glyph images makes up a font. Glyphs can be construed as the basic units of organization of the visual rendering of text, just as characters are the basic unit of organization of encoded text.

[S] [I] Specifications and software MUST NOT assume a one-to-one mapping between character codes and units of displayed text.

See the appendix B Examples of Characters, Keystrokes and Glyphs for examples of the complexities of character to glyph mapping.

Some scripts, in particular Arabic and Hebrew, are written from right to left. Text including characters from these scripts can run in both directions and is therefore called bidirectional text. The Unicode Standard [Unicode] requires that characters be stored and interchanged in logical order. [S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order.

In the presence of bidirectional text, two possible selection modes must be considered. The first is logical selection mode, which selects all the characters logically located between the end-points of the user's mouse gesture. Here the user selects from between the first and second letters of the second word to the middle of the number. Logical selection looks like this:

In memory
On screen

It is a consequence of the bidirectionality of the text that a single, continuous logical selection in memory results in a discontinuous selection appearing on the screen. This discontinuity, as well as the somewhat unintuitive behavior of the cursor, makes some users prefer a visual selection mode, which selects all the characters visually located between the end-points of the user's mouse gesture. With the same mouse gesture as before, we now obtain:

In memory
On screen

In this mode, a single visual selection range results in two logical ranges, which have to be accommodated by protocols, APIs and implementations.

[S] Specifications of protocols and APIs that involve selection of ranges SHOULD provide for discontiguous selections, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs.

3.1.4 Units of input

In keyboard input, it is not always the case that keystrokes and input characters correspond one-to-one. A limited number of keys can fit on a keyboard. Some keyboards will generate multiple characters from a single keypress. In other cases ('dead keys') a key will generate no characters, but affect the results of subsequent keypresses. Many writing systems have far too many characters to fit on a keyboard and must rely on more complex input methods, which transform keystroke sequences into character sequences. Other languages may make it necessary to input some characters with special modifier keys. See B Examples of Characters, Keystrokes and Glyphs for examples of non-trivial input.

[S] [I] Specifications and software MUST NOT assume that a single keystroke results in a single character, nor that a single character can be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world.

3.1.5 Units of collation

String comparison as used in sorting and searching is based on units which do not in general have a one-to-one relationship to encoded characters. Such string comparison can aggregate a character sequence into a single collation unit with its own position in the sorting order, can separate a single character into multiple collation units, and can distinguish various aspects of a character (case, presence of diacritics, etc.) to be sorted separately (multi-level sorting).

In addition, a certain amount of pre-processing may also be required, and in some languages (such as Japanese and Arabic) sort order may be governed by higher order factors such as phonetics or word roots. Collation methods may also vary by application.

[S] [I] Software that sorts or searches text for users MUST do so on the basis of appropriate collation units and ordering rules for the relevant language and/or application.

Note that, where searching or sorting is done dynamically, particularly in a multilingual environment, the 'relevant language' should be determined to be that of the current user, and may thus differ from user to user. [S] [I] Software that allows users to sort or search text SHOULD allow the user to select alternative rules for collation units and ordering.

[S] [I] When sorting and searching in the context of a particular language, it MUST be possible to deal gracefully with strings being compared that contain Unicode characters not normally associated with that language. A default collation order for all Unicode characters can be obtained from ISO/IEC 14651 [ISO/IEC 14651] or from Unicode Technical Report #10, the Unicode Collation Algorithm [UTR #10]. This default ordering can be used in conjunction with rules tailored for a particular locale to ensure a predictable ordering and comparison of strings, whatever characters they include.

3.1.6 Units of storage

Computer storage and communication rely on units of physical storage and information interchange, such as bits and bytes (also known as octets, as nowadays the word bytes is generally considered to mean 8-bit bytes). A frequent error in specifications and implementations is the equating of characters with units of physical storage. The mapping between characters and such units of storage is actually quite complex, and is discussed in the next section, 3.2 Digital Encoding of Characters.

[S] [I] Specifications and software MUST NOT assume a one-to-one relationship between characters and units of physical storage.

3.1.7 Summary

The term character is used differently in a variety of contexts and often leads to confusion when used outside of these contexts. In the context of the digital representations of text, a character can be defined informally as a small logical unit of text. Text is then defined as sequences of characters. While such an informal definition is sufficient to create or capture a common understanding in many cases, it is also sufficiently open to create misunderstandings as soon as details start to matter. In order to write effective specifications, protocol implementations, and software for end users, it is very important to understand that these misunderstandings can occur.

[S] When specifications use the term 'character' it MUST be clear which of the possible meanings they intend. [S] Specifications SHOULD avoid the use of the term 'character' if a more specific term is available.

3.6.1 Mandating a unique character encoding

Mandating a unique character encoding is simple, efficient, and robust. There is no need for specifying, producing, transmitting, and interpreting encoding tags. At the receiver, the encoding will always be understood. There is also no ambiguity as to which encoding to use if data is transferred non-electronically and later has to be converted back to a digital representation. Even when there is a need for compatibility with existing data, systems, protocols and applications, multiple encodings can often be dealt with at the boundaries or outside a protocol, format, or API. The DOM [DOM Level 1] is an example of where this was done. The advantages of choosing a unique encoding become more important the smaller the pieces of text used are and the closer to actual processing the specification is.

[S] When a unique encoding is mandated, the encoding MUST be UTF-8, UTF-16 or UTF-32. [S] If a unique encoding is mandated and compatibility with US-ASCII is desired, UTF-8 (see [RFC 2279]) is RECOMMENDED. In other situations, such as for APIs, UTF-16 or UTF-32 may be more appropriate. Possible reasons for choosing one of these include efficiency of internal processing and interoperability with other processes.

3.6.2 Character encoding identification

The MIME Internet specification [MIME] provides a good example of a mechanism for character encoding identification. The MIME charset parameter definition is intended to supply sufficient information to uniquely decode the sequence of bytes of the received data into a sequence of characters. The values are drawn from the IANA charset registry [IANA].

[S] Specifications SHOULD avoid using the terms 'character set' and 'charset' to refer to a character encoding, except when the latter is used to refer to the MIME charset parameter or its IANA-registered values. The terms 'character encoding', 'character encoding form' or 'character encoding scheme' are RECOMMENDED.

The IANA charset registry is the official list of names and aliases for character encodings on the Internet.

[S] If the unique encoding approach is not taken, specifications SHOULD mandate the use of the IANA charset registry names, and in particular the names identified in the registry as 'MIME preferred names', to designate character encodings in protocols, data formats and APIs. [S] The 'x-' convention for unregistered character encoding names SHOULD NOT be used, having led to abuse in the past. ('x-' was used for character encodings that were widely used, even long after there was an official registration.) [I] [C] Content and software that label text data MUST use one of the names mandated by the appropriate specification (e.g. the XML specification when editing XML text) and SHOULD use the MIME preferred name of an encoding to label data in that encoding. [I] [C] An IANA-registered charset name MUST NOT be used to label text data in an encoding other than the one identified in the IANA registration of that name.

[S] If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible encodings and SHOULD choose at least one of UTF-8 or UTF-16 as mandated encoding forms (encoding forms that MUST be supported by implementations of the specification). [S] Specifications MAY define either UTF-8 or UTF-16 as a default encoding form (or both if they define suitable means of distinguishing them), but they MUST NOT use any other character encoding as a default. [S] Specifications MUST NOT propose the use of heuristics to determine the encoding of data.

[I] Receiving software MUST determine the encoding of data from available information according to appropriate specifications. [I] When an IANA-registered charset name is recognized, receiving software MUST interpret the received data according to the encoding associated with the name in the IANA registry. [I] When no charset is provided receiving software MUST adhere to the default encoding(s) specified in the specification.

[I] Receiving software MAY recognize as many encodings (names and aliases) as appropriate. A field-upgradeable mechanism may be appropriate for this purpose. Certain encodings are more or less associated with certain languages (e.g. Shift-JIS with Japanese); trying to support a given language or set of customers may mean that certain encodings have to be supported. The encodings that need to be supported may change over time. This document does not give any advice on which encoding may be appropriate or necessary for the support of any given language.

[I] Software MUST completely implement the mechanisms for character encoding identification and SHOULD implement them in such a way that they are easy to use (for instance in HTTP servers). [I] On interfaces to other protocols, software SHOULD support conversion between Unicode encoding forms as well as any other necessary conversions.

[C] Content MUST make use of available facilities for character encoding identification by always indicating character encoding; where the facilities offered for character encoding identification include defaults (e.g. in XML 1.0 [XML 1.0]), relying on such defaults is sufficient to satisfy this identification requirement.

Because of the layered Web architecture (e.g. formats used over protocols), there may be multiple and at times conflicting information about character encoding. [S] Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding. [I] [C] Software and content MUST carefully follow conflict-resolution mechanisms where there is multiple or conflicting information about character encoding.

3.6.3 Private use code points

Unicode designates certain ranges of code points for private use: the Private Use Area (U+E000-F8FF) and planes 15 and 16 (U+F0000-FFFFD and U+100000-10FFFD). These code points are guaranteed to never be allocated to standard characters, and are available for use by private agreement between a producer and a recipient. However, their use is strongly discouraged, since private agreements do not scale on the Web. Code points from different private agreements may collide, and a private agreement and therefore the meaning of the code points can quickly get lost.

[S] Specifications MUST NOT define any assignments of private use code points. [S] Conformance to a specification MUST NOT require the use of private use area characters. [S] Specifications SHOULD NOT provide mechanisms for agreement on private use code points between parties and MUST NOT require the use of such mechanisms. [S] [I] Specifications and implementations SHOULD be designed in such a way as to not disallow the use of private use code points by private arrangement. As an example, XML does not disallow the use of private use code points.

[S] Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters.

4.1.1 Why do we need character normalization?

Text in computers can be encoded in one of many encodings. In addition, some encodings allow multiple representations for the 'same' string and Web languages have escape mechanisms that introduce even more equivalent representations. For instance, in ISO 8859-1 the letter 'ç' can only be represented as the single character E7 'ç', in a Unicode encoding it can be represented as the single character U+00E7 'ç' or the sequence U+0063 'c' U+0327 '¸', and in HTML it could be additionally represented as ç or ç or ç.

There are a number of fundamental operations that are sensitive to these multiple representations: string matching, indexing, searching, sorting, regular expression matching, selection, etc. In particular, the proper functioning of the Web (and of much other software) depends to a large extent on string matching. Examples of string matching abound: parsing element and attribute names in Web documents, matching CSS selectors to the nodes in a document, matching font names in a style sheet to the names known to the operating system, matching URI pieces to the resources in a server, matching strings embedded in an ECMAscript program to strings typed in by a Web form user, matching the parts of an XPath expression (element names, attribute names and values, content, etc.) to what is found in an instance, etc.

String matching is usually taken for granted and performed by comparing two strings byte for byte, but the existence on the Web of multiple character representations means that it is actually non-trivial. Binary comparison does not work if the strings are not in the same encoding (e.g. an EBCDIC style sheet being directly applied to an ASCII document, or a font specification in a Shift-JIS style sheet directly used on a system that maintains font names in UTF-16) or if they are in the same encoding but show variations allowed for the 'same' string by the use of combining characters or by the constructs of the Web language.

Incorrect string matching can have far reaching consequences, including the creation of security holes. Consider a contract, encoded in XML, for buying goods: each item sold is described in an artículo element; unfortunately, "artículo" is subject to different representations in the character encoding of the contract. Suppose that the contract is viewed and signed by means of a user agent that looks for artículo elements, extracts them (matching on the element name), presents them to the user and adds up their prices. If different instances of the artículo element happen to be represented differently in a particular contract, then the buyer and seller may see (and sign) different contracts if their respective user agents perform string identity matching differently, which is fairly likely in the absence of a well-defined specification for string matching. The absence of a well-defined specification would also mean that there would be no way to resolve the ensuing contractual dispute.

Solving the string matching problem involves normalization, which in a nutshell means bringing the two strings to be compared to a common, canonical encoding prior to performing binary matching. (For additional steps involved in string matching see 6 String Identity Matching.)

4.1.2 The choice of early uniform normalization

There are options in the exact way normalization can be used to achieve correct behaviour of normalization-sensitive operations such as string matching. These options lie along two axes:

The first axis is a choice of when normalization occurs: early (when strings are created) or late (when strings are compared). The former amounts to establishing a canonical encoding for all data that is transmitted or stored, so that it doesn't need any normalization later, before being used. The latter is the equivalent of mandating 'smart' compare functions, which will take care of any encoding differences.

This document specifies early normalization. The reasons for that choice are manifold:

• Almost all legacy data as well as data created by current software is normalized (if using NFC).

• The number of Web components that generate or transform text is considerably smaller than the number of components that receive text and need to perform matching or other processes requiring normalized text.

• Current receiving components (browsers, XML parsers, etc.) implicitly assume early normalization by not performing or verifying normalization themselves. This is a vast legacy.

• Web components that generate and process text are in a much better position to do normalization than other components; in particular, they may be aware that they deal with a restricted repertoire only, which simplifies the process of normalization.

• Not all components of the Web that implement functions such as string matching can reasonably be expected to do normalization. This, in particular, applies to very small components and components in the lower layers of the architecture.

• Forward-compatibility issues can be dealt with more easily: less software needs to be updated, namely only the software that generates newly introduced characters.

• It improves matching in cases where the character encoding is partly undefined, such as URIs [RFC 2396] in which non-ASCII bytes have no defined meaning.

• It is a prerequisite for comparison of encrypted strings (see [CharReq], section 2.7).

The second axis is a choice of canonical encoding. This choice needs only be made if early normalization is chosen. With late normalization, the canonical encoding would be an internal matter of the smart compare function, which doesn't need any wide agreement or standardization.

By choosing a single canonical encoding, it is insured that normalization is uniform throughout the web. Hence the two axes lead us to the name 'early uniform normalization'.

4.1.3 The choice of Normalization Form C

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms [UTR #15]). These forms differ in 1) whether they normalize towards decomposed characters (NFD, NFKD) or precomposed characters (NFC, NFKC) and 2) whether they normalize away compatibility distinctions (NFKD, NFKC) or not (NFD, NFC).

For use on the Web, it is important not to lose the so-called compatibility distinctions, which may be important (see [UXML] for a discussion). The 'K' normalization forms are therefore excluded. Among the remaining two forms, NFC has the advantage that almost all legacy data (if transcoded trivially, one-to-one) as well as data created by current software is already in this form; NFC also has a slight compactness advantage and a better match to user expectations with respect to the character vs grapheme issue. This document therefore chooses NFC as the base for Web-related text normalization.

For a list of programming resources related to normalization, see D Resources for Normalization.

4.2.1 Unicode-normalized text

Text is, for the purposes of this specification, Unicode-normalized if it is in a Unicode encoding form and is in Unicode Normalization Form C, according to a version of Unicode Standard Annex #15: Unicode Normalization Forms [UTR #15] at least as recent as the oldest version of Unicode that contains all the characters actually present in the text, but no earlier than version 3.2 [Unicode 3.2].

4.2.2 Include-normalized text

Markup languages, style languages and programming languages often offer facilities for including a piece of text inside another. An include is an instance of a syntactic device specified in a language to include an entity at the position of the include, replacing the include itself. Examples of includes are entity references in XML, @import rules in CSS and the #include preprocessor statement in C/C++. Character escapes are a special case of includes where the included entity is predetermined by the language.

Text is include-normalized if:

1. the text is Unicode-normalized and does not contain any character escapes or includes whose expansion would cause the text to become no longer Unicode-normalized; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

4.2.3 Fully-normalized text

Formal languages define constructs, which are identifiable pieces, occuring in instances of the language, such as comments, identifiers, element tags, processing instructions, runs of character data, etc. During the normal processing of include-normalized text, these various constructs may be moved, removed (e.g. removing comments) or merged (e.g. merging all the character data within an element as done by the string() function of XPath), creating opportunities for text to become denormalized. The software performing those operations then has to re-normalize the result, which is a burden. One way to avoid such denormalization is to make sure that the various important constructs never begin with a character such that appending that character to a normalized string can cause the string to become denormalized. A composing character is a character that is one or both of the following:

1. the second character in the canonical decomposition mapping of some primary composite (as defined in D3 of [UTR #15]), or

2. of non-zero canonical combining class (as defined in [Unicode]).

Please consult Appendix C Composing Characters for a discussion of composing characters, which are not exactly the same as Unicode combining characters.

Text is fully-normalized if:

1. the text is in a Unicode encoding form, is include-normalized and none of the constructs comprising the text begin with a composing character or a character escape representing a composing character; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.

Identification of the constructs that should be prohibited from beginning with a composing character (the relevant constructs) is language-dependent. As specified in 4.4 Responsibility for Normalization, it is the responsibility of the specification for a language to specify exactly what constitutes a relevant construct. This may be done by specifying important boundaries, taking into account which operations would benefit the most from being protected against denormalization. The relevant constructs are then defined as the spans of text between the boundaries. At a minimum, for those languages which have these notions, the important boundaries are entity (include) boundaries as well as the boundaries between most markup and character data. Many languages will benefit from defining more boundaries and therefore finer-grained full-normalization constructs.

4.3.1 General examples

The string suçon (U+0073 U+0075 U+00E7 U+006F U+006E) encoded in a Unicode encoding form, is Unicode-normalized, include-normalized and fully-normalized. The same string encoded in a legacy encoding for which there exists a normalizing transcoder would be both include-normalized and fully-normalized but not Unicode-normalized (since not in a Unicode encoding form).

In an XML or HTML context, the string suçon is also include-normalized, fully-normalized and, if encoded in a Unicode encoding form, Unicode-normalized. Expanding ç yields suçon as above, which contains no replaceable combining sequence.

The string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E), where U+0327 is the COMBINING CEDILLA, encoded in a Unicode encoding form, is neither Unicode-normalized (since the combining sequence 'c¸' (U+0063 U+0327) should be replaced by the precomposed 'ç' (U+00E7). As a consequence this string is neither include-normalized (since in a Unicode encoding form but not Unicode-normalized) nor fully-normalized (since not include-normalized). Note however that the string sub¸on (U+0073 U+0075 U+0062 U+0327 U+006F U+006E) in a Unicode encoding is Unicode-normalized since there is no precomposed form of 'b' plus cedilla. It is also include-normalized and fully-normalized.

In plain text the string suçon is Unicode-normalized, since plain text doesn't recognise that ̧ represents a character in XML or HTML and considers it just a sequence of non-replaceable characters.

In an XML or HTML context, however, expanding ̧ yields the string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E) which is not include-normalized ('c¸' is replaceable by 'ç'). As a consequence the string is neither include-normalized nor fully-normalized. As another example, if the entity reference &word-end; refers to an entity containing ¸on (U+0327 U+006F U+006E), then the string suc&word-end; is not include-normalized for the same reasons.

In an XML or HTML context, expanding ̧ in the string sub̧on yields the string sub¸on which is Unicode-normalized since there is no precomposed character for 'b cedilla' in NFC. This string is therefore also include-normalized. Similarly, the string sub&word-end; (with &word-end; as above) is include-normalized, for the same reasons.

In an XML or HTML context, the strings ¸on (U+0327 U+006F U+006E) and ̧on are not fully-normalized, as they begin with a composing character (after expansion of the character escape for the second). However, both are Unicode-normalized (if expressed in a Unicode encoding) and include-normalized.

The following table consolidates the above examples.

4.3.2 Examples of XML in a Unicode encoding

Here is another summary table, with more examples but limited to XML in a Unicode encoding. The following list describes what the entities contain and special character usage. Normalised forms are indicated using 'Y'. There is no precomposed 'b with cedilla' in NFC.

• "ç" LATIN SMALL LETTER C WITH CEDILLA

• "&cedilla;" CEDILLA (combining)

• "&c;" LATIN SMALL LETTER C

• "&b;" LATIN SMALL LETTER B

• "¸" CEDILLA (combining)

• "/" (immediately before 'on' in last example) COMBINING LONG SOLIDUS OVERLAY

4.3.3 Examples of restrictions on the use of combining characters

Include-normalization and full-normalization create restrictions on the use of combining characters. The following examples discuss various such potential restrictions and how they can be addressed.

Full-normalization prevents the markup of an isolated combining mark, for example for styling it differently from its base character (Benoi<span style='color: blue'>^</span>t, where '^' represents a combining circumflex). However, the equivalent effect can be achieved by assigning a class to the accents in an SVG font or using equivalent technology. View an example using SVG (SVG-enabled browsers only).

Full-normalization prevents the use of entities for expressing composing characters. This limitation can be circumvented by using character escapes or by using entities representing complete combining character sequences. With appropriate entity definitions, instead of A´, write Á (or better, use 'Á' directly).