Overview

• Overview of legacy character sets
• 7-bit: ASCII
• 8-bit: ISO-8859
• 16-bit: Chinese-Japanese-Korean
• Introduction to Unicode
• Philosophy
• History
• Design
• Relationship with ISO-10646

Character Set and Character Encoding

Character Repertoire

a finite set of characters

Coded Character Set

a finite set of characters, where each character is assigned an unique and specific number

Character Encoding

scheme to encode a series of characters into a binary data stream

Legacy Character Set: ASCII (1)

• The "mother" of all modern character sets
• Developed in the 1960s for telecommunication and computer use
• 7-bits ⇒ 128 characters
• 94 visible characters + "Space" + 32 control characters
• Only perfect for Latin, Swahili, Hawaiian and American English

Legacy Character Set: ASCII (2)

ASCII characters

Legacy Character Set: ISO-646 - the international ASCII

• Framework for national variants of ASCII
• Standardized by ISO
• Allow infrequently used characters of ASCII to be replaced: # $ @ [ \ ] ^ ` { | } ~
• German variant (ISO-646-DE): ä ö ü Ä Ö Ü ß
• Japanese variant (ISO-646-JP): \ ⇒ ¥, ~ ⇒〜
• More than 25 variants: Danish, Spanish, French, Italian, Korean, Yugoslavia, ...
• No mixing allowed, only one language covered at the same time
• Still 7-bit, limited to 94 visible characters

Legacy Character Set: ISO-8859 for Europe (1)

• ASCII usually stored in 8-bit (byte) units, 8th bit unused
• Use 8th-bit: 256 characters
• 4 character sets defined by the ECMA: West/East/North/South Europe
• Contain characters for multiple languages
• Superset of ASCII: new characters at 0x80 - 0xff,
  0x00 - 0x7f unchanged

Legacy Character Set: ISO-8859 for Europe (2)

• Adopted by ISO as ISO-8859
• Latin-1 (West Europe) widely used as ISO-8859-1
• More character sets added over time: Cyrillic, Arabic, Greek, Hebrew, Turkish ...
• ISO-8859-15 adds € symbol, successor of ISO-8859-1

Legacy Character Set: ISO-8859 for Europe (3)

Legacy Character Set: ISO-8859 for Europe (4)

• Internet not completely 8-bit safe
• 8th bit might be dropped in transit
• Need to encode characters to look like ASCII
• "Quoted-Printable" encoding
• Multiple ASCII characters for one 8-bit character
• Hexadecimal value written in ASCII
• ä (0xE4) ⇒ =E4
• = (0x3d) ⇒ =3D

Legacy Character Sets : Chinese, Japanese, Korean (1)

• CJK languages use ideographs
• Several thousand ideographs needed for daily life
• 256 characters insufficient ⇒ 16-bit character sets
• JIS X 0208, first multi-byte character set for Japanese (1976)
• Organized as 94 rows with 94 cells each, total of 8,836 code pints, compatibility with ASCII
• Korean and Chinese design own JIS X 0208-like character sets
• China: GB 2312 (mainland), CNS 11643 & Big5 (Taiwan)
• Korea: KS C 5601 (official), UHC (vendor)

Legacy Character Sets: Chinese, Japanese, Korean (2)

JIS X 0208 characters (partial)

Legacy Character Sets: Chinese, Japanese, Korean (3)

• Again, the characters need to be encoded
• Several schemes are used widely
• Encodings for Japanese: raw JIS (Internet), EUC-JP (UNIX), Shift-JIS (Windows, Mac)
• No method to detect which encoding is used
• Systems need to convert between encodings

Legacy Character Encoding: ISO-2022 (1)

• "Information technology -- Character code structure and extension techniques"
• One encoding to rule them all
• Framework for Multilingualization (m17n)
• State-full encoding using escape-sequences
• Assign 4 character sets to G0..G3
• Switch between G0..G3

Legacy Character Encoding: ISO-2022 (2)

Legacy Character Encoding: ISO-2022 (3)

• Very complex to implement
• Application must have knowledge of used character sets
• Used by m17n-ed Emacs internally
• Whole text must be known to process correctly
• Simplified variants: ISO-2022-{JP,KO,CN} (fixed character sets at G0..G3)

Chaotic World

• Many incompatible character sets and encodings
• No simple way to use multiple languages at the same time
• E.g: Encoding must be set manually in Web-browsers
• Possible solution: Unicode

Unicode: The Universal Character Set

Universal

Cover most characters world-wide

Efficient

Must be simple to implement, efficient processing

Uniform

Sorting, searching and display without special exception rules

Unambiguous

Any given code value always represents same character

Unicode: History

1986-88	basic design of Unicode at Xerox, Apple
Dec. 1987	first use of term "Unicode"
1988,89	Kanji-Unification work
1990	major players like Sun, IBM join
Jan 1991	founding of Unicode, Inc.
Oct 1991	Unicode 1.0
Jun 1993	Unicode 1.1 - merge with ISO/IEC 10646
Jul 1996	Unicode 2.0 - extention of character space
May 1998	Unicode 2.1.2 - add Euro symbol
Sep 1999	Unicode 3.0 - clarify encodings and properties
Mar 2001	Unicode 3.1 - add many more characters
Mar 2002	Unicode 3.2 - conformance tight up

Unicode: Number of Characters

• Unicode 1.1: 34,203 (1991)
• Unicode 3.2: 95,156 (2002)

Unicode: characters, not glyphs

• Unicode does not encode glyphs but characters.
• Distinction not by visual appearance, but semantic units

LATIN CAPITAL LETTER A

Unicode: 16-bit fixed-length

• Each character has a unique and specific number
• 16-bit : 2^16 = 65,535 characters possible
• Notation : U+0000 - U+FFFF
• Unicode 2.0: extention for 17 planes with 2^16 characters each
• Plane 00: Basic Multilingual Plane (BMP)

Unicode: Universal Character Set (1)

• Include characters for as many languages as possible
• Organization by blocks of character type

Unicode: Universal Character Set (2)

• Space between blocks: for future allocation from same language
• Blocks divided into sub-blocks based on origin language

Unicode: Universal Character Set (3)

• "Standard" scripts covered
• Latin, Greek, Cyrillic
• Kanji, Hiragana, Katakana, Hangul
• Arabic, Hebrew
• "Exotic" scripts covered
  • Cherokee, Canadian Aboriginal, Runic
• Planned future additions
• Hieroglyphics, Javanese, Old Hungarian
• Dismissed proposals
• Clingon (Star Trek)
• Tengwar (Elfish script in Lord of the Rings)

Unicode: Logical Order (1)

• Some languages are left-to-right, others right-to-left
• Characters are exclusively stored in logical order
• Logical order: order of reading, not neccersary visual order
• Character direction determined with Character Property
• More complex rendering, but simple searching and sorting

Unicode: Logical Order (2)

Display Order: ABCD:אבגד

Logical Order:

A

B

C

D

:

א

ב

ג

ד

Unicode: Character Properties

• Characters have properties that define their semantic
• Normative properties: must be implemented for conformance
• Informative properties: implementation not required
• Example Properties:
• Alphabetic
• Case
• Directionality
• Numeric Value
• Surrogate
• Decomposition
• Mirrored

Unicode: Dynamic Composition

• Many accented characters included in Unicode: Ü,Ç, ǻ
• Can also be represented using Dynamic Composition
• 1 base character + n combining characters
• Creation of new/not-included characters
• a + ̈ ⇒ ä
• a + ̊ + ́ ⇒ ǻ
• C + ̧ ⇒ Ç

Unicode: Decomposition Mappings

• Dynamic Composition: multiple ways to encode a single character
• ǻ could be ǻ, å + ́ or a + ̊ + ́
• Difficult to search and sort
• Normalization through Decomposition Mappings
• Character Property specifies how a character can be disassembled

Unicode: Unification (1)

• Many languages share common scripts (e.g: Latin alphabet)
• characters with same appearance unified into single code value
• E.g: symbol for seconds, feet and prime unified to PRIME (′)
• More than 130,000 CJK ideographs unified to less than 30,000
• But, Unification often problematic

Unicode: Unification (2)



• Small difference for Europeans and Americans, huge difference for East Asians
• Visual variation maybe guess-able, often simply wrong
• Rendering dependent on language
• No single Unicode font would be sufficient

Unicode: Unification (2)

Locale	Language	U+5E73	U+76F4
ja	Japanese	平	直
ko	Korean	平	直
zh-TW	Chinese (Taiwan)	平	直
zh-CN	Chinese (China)	平	直

• Early versions unusable for East Asian languages
• Language Tags added to Unicode 3.0
• Variants and additional characters being added into extra planes, starting with Unicode 3.1

Unicode: Surrogate Pairs

• 16-bit (65,535) characters not enough
• How to fit more characters ?
• Two 16-bit values = 1 character
• 2048 "Surrogate" characters reserved, 1024 high, 1024 low
• 1 high-surrogate followed by 1 low surrogate
• 2^20 additional characters: U+0100 - U+10FFFF
• Algorithm (UTF-16) used to computer real character code
• U+D801 U+DC01 ⇒ U+010401

ISO/IEC 10646

• "Information technology -- Universal Multiple-Octet Coded Character Set (UCS)"
• 4 octets per character (31-bit)
  >= 2*10^9 characters
• Character-by-character compatibility with Unicode
• Joint-development between Unicode and ISO
• Win-Win partnership
• ISO better usable as a reference for national standards
• Unicode has industry-support and feedback

Conclusion

• Unicode most practical solution to i18n/m17n
• More and more applications support Unicode
• Evolving into really usable standard
• Some contradictions to own design principles
• Applications still need to support legacy character sets
• "Unicode-only" philosophy will fail
• Unicode: practical tool, but not THE ultimate tool

Thank you

Any Questions?