<html><title>data</title><body BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#330088" ALINK="#FF0044"><H1>Hello World<br>or<br>¿ALPHA¿¿MU¿¿ALPHA ¿¿¿MUEPSILON<br>or<br>¿¿¿¿¿ ¿¿</H1><DL><DD><I>Rob Pike<br>Ken Thompson<br><br> <br>rob,ken@plan9.bell-labs.com<br></I></DL><DL><DD><H4>ABSTRACT</H4><DL><DT><DT> <DD>NOTE:<I> Originally appeared, in a slightly different form, inProc. of the Winter 1993 USENIX Conf.,pp. 43-50,San Diego</I><DT> <DD></dl><br>Plan 9 from Bell Labs has recently been converted from ASCIIto an ASCII-compatible variant of the Unicode Standard, a 16-bit character set.In this paper we explain the reasons for the change,describe the character set and representation we chose,and present the programming models and software changesthat support the new text format.Although we stopped short of full internationalization­forexample, system error messages are in Unixese, not Japanese­webelieve Plan 9 is the first system to treat the representationof all major languages on a uniform, equal footing throughout all itssoftware.</DL><H4>Introduction</H4><P>The world is multilingual but most computer systemsare based on English and ASCII.The first release of Plan 9 [Pike90], a new distributed operatingsystem from Bell Laboratories, seemed a good occasionto correct this chauvinism.It is easier to make such deep changes when building new systems thanby refitting old ones.</P><P>The ANSI C standard [ANSIC] contains some guidance on the matter of`wide' and `multi-byte' characters but falls far short ofsolving the myriad associated problems.We could find no literature on how to convert a<I>system</I>to larger character sets, although some individual<I>programs</I>had been converted.This paper reports what we discovered as weexplored the problem of representing multilingualtext at all levels of an operating system,from the file system and kernel throughthe applications and up to the window systemand display.</P><P>Plan 9 has not been `internationalized':its manuals are in English,its error messages are in English,and it can display text that goes from left to right only.But before we can address these other problems,we need to handle, uniformly and comfortably,the textual representation of all the major written languages.That subproblem is richer than we had anticipated.</P><H4>Standards</H4><P>Our first step was to select a standard.At the time (January 1992),there were only two viable options:ISO 10646 [ISO10646] and Unicode [Unicode].The documents describing both proposals were still in the draft stage.</P><P>The draft of ISO 10646 was notvery attractive to us.It defined a sparse set of 32-bit characters,which would behard to implementand have punitive storage requirements.Also, the draft attempted tomollify national interests by allocating16-bit subspaces to national committeesto partition individually.The suggested mode of use was to``flip'' between separate nationalstandards to implement the international standard.This did not strike us as a sound basis for a character set.As well, transmitting 32-bit values in a byte stream,such as in pipes, would be expensive and hard to implement.Since the standard does not define a byte order for suchtransmission, the byte stream would also have to carrystate to enable the values to be recovered.</P><P>The Unicode Standard is a proposal by a consortium of mostly Americancomputer companies formedto protest the technicalfailings of ISO 10646.It defines a uniform 16-bit code based on theprinciple of unification:two characters are the same if they look thesame even though they are from differentlanguages.This principle, called Han unification,allows the large Japanese, Chinese, and Koreancharacter sets to be packed comfortably into a 16-bit representation.</P><P>We chose the Unicode Standard for its technical merits and because itscode space was better defined.Moreover,the Unicode Consortium was derailing theISO 10646 standard.(Now, in 1995,ISO 10646 is a standardwith one 16-bit group defined,which is almost exactly the Unicode Standard.As most people expected, the two standards bodiesreached a détente andISO 10646 and Unicode represent the same character set.)</P><P>The Unicode Standard defines an adequate character setbut an unreasonable representation.It states that all charactersare 16 bits wide and are communicated and stored in16-bit units.It also reserves a pair of characters(hexadecimal FFFE and FEFF) to detect byte orderin transmitted text, requiring state in the byte stream.(The Unicode Consortium was thinking of files, not pipes.)To adopt this encoding,we would have had to convert all text goinginto and out of Plan 9 between ASCII and Unicode, which cannot be done.Within a single program, in command of all its input and output,it is possible to define characters as 16-bit quantities;in the context of a networked system withhundreds of applications on diverse machinesby different manufacturers,it is impossible.</P><P>We needed a way to adapt the Unicode Standard to the tools-and-pipesmodel of text processing embodied by the Unix system.To do that, weneeded an ASCII-compatible textualrepresentation of Unicode characters for transmissionand storage.In the draft ISO standard there was an informative(non-required)Annexcalled UTFthat provided a byte stream encodingof the 32-bit ISO code.The encoding uses multibyte sequences composedfrom the 190 printable characters of Latin-1to represent character values largerthan 159.</P><P>The UTF encoding has several good properties.By far the most important is thata byte in the ASCII range 0-127 representsitself in UTF.Thus UTF is backward compatible with ASCII.</P><P>UTF has other advantages.It is a byte encoding and istherefore byte-order independent.ASCII control characters appear in the byte streamonly as themselves, never as an element of a sequenceencoding another character,so newline bytes separate lines of UTF text.Finally, ANSI C's<TT>strcmp</TT>function applied to UTF strings preserves the ordering of Unicode characters.</P><P>To encode and decode UTF is expensive (involving multiplication,division, and modulo operations) but workable.UTF's major disadvantage is that the encodingis not self-synchronizing.It is in general impossible to find the characterboundaries in a UTF string without reading fromthe beginning of the string, although in practicecontrol characters such as newlines,tabs, and blanks provide synchronization points.</P><P>In August 1992,X-Open circulated a proposal for another UTF-likebyte encoding of Unicode characters.Their major concern was that an embedded characterin a file name(in particular a slash)could be part of an escape sequence in UTF andtherefore confuse a traditional file system.Their proposal would allow all 7-bit ASCII charactersto represent themselves<I>and only themselves</I>in text.Multibyte sequences would contain only characterswith the high bit set.We proposed a modification to the new UTF thatwould address our synchronization problem.Our proposal, which was originally known informally as UTF-2 and FSS-UTF,is now referred to as UTF-8 and has been approved by ISO to becomeAnnex P to ISO 10646.</P><P>The model for text in Plan 9 is chosen from thesethree standards*:</P><DL><DT><DT> <DD>NOTE:<I> * ``That's the nice thing about standards­there's so many to choose from.'' - Andy Tannenbaum (no, the other one)</I><DT> <DD></dl><br>the Unicode character set encoded as a byte stream byUTF-8, from(soon to be) Annex P of ISO 10646.Although this mixture may seem like a precarious position for us to adopt,it is not as bad as it sounds.ISO 10646 and the Unicode Standard have converged,other systems such as Linux have adopted the same character set and encoding,and the general feeling seems to be that Unicode and UTF-8 will be accepted as the wayto exchange text between systems.The prognosis for wide acceptance is good.<P>There are a couple of aspects of the Unicode Standard we have not faced.One is the issue of right-to-left text such as Hebrew or Arabic.Since that is an issue of display, not representation, we believewe can defer that problem for the moment without affecting ourability to solve it later.Another issue is diacriticals and `combining characters',which cause overstriking of multiple Unicode characters.Although necessary for some scripts, such as Thai, Arabic, and Hebrew,such characters confuse the issues for Latin languages because theygenerate multiple representations for accented characters.ISO 10646 describes three levels of implementation;in Plan 9 we decided not to address the issue.Again, this can be labeled as a display issue and its finer points are still being debated,so we felt comfortable deferring. Mañana.</P><P>Although we converted Plan 9 in the altruistic interests ofserving foreign languages, we have found the large characterset attractive for other reasons. The Unicode Standard includes manycharacters­mathematical symbols, scientific notation,more general punctuation, and more­that we now usedaily in our work. We no longer test our imaginationsto find ways to include non-ASCII symbols in our text;why type<TT>:-)</TT>when you can use the character ¿?Most compelling is the ability to absorb documentsand data that contain non-ASCII characters; our browser for theOxford English Dictionarylets us see the dictionary as it really is, with pronunciationin the IPA font, foreign phrases properly rendered, and so on,<I>in plain text.</I></P><P>In the rest of this paper, except whenstated otherwise, the term `UTF' refers to the UTF-8 encodingof Unicode characters as adopted by Plan 9.</P><H4>C Compiler</H4><P>The first program to be converted to UTFwas the C Compiler.There are two levels of conversion.On the syntactic level,input to the C compileris UTF; on the semantic level,the C language needs to definehow compiled programs manipulatethe UTF set.</P><P>The syntactic part is simple.The ANSI C language standard defines thesource character set to be ASCII.Since UTF is backward compatible with ASCII,the compiler needs little change.The only places where a larger character setis allowed are in character constants, strings, and comments.Since 7-bit ASCII characters can represent onlythemselves in UTF,the compiler does not have to be careful while lookingfor the termination of a string or comment.</P><P>The Plan 9 compiler extends ANSI C to treat any Unicodecharacter with a value outside of the ASCII range asan alphabetic.To a Greek programmer or an English mathematician,ALPHA is a sensible and now valid variable name.</P><P>On the semantic level, ANSI C allows,but does not tie down,the notion of a<I>wide character</I>and admits string and character constantsof this type.We chose the wide character type to be<TT>unsigned</TT><TT>short</TT>.In the libraries, the word<TT>Rune</TT>is defined by a<TT>typedef</TT>to be equivalent to<TT>unsigned</TT><TT>short</TT>and isused to signify a Unicode character.</P><P>There are surprises; for example:<DL><DT><DD><TT><PRE>L'x' is 120'x' is 120L'ÿ' is 255'ÿ' is -1, stdio EOF (if char is signed)L'ALPHA' is 945'ALPHA' is illegal</PRE></TT></DL>In the string constants,<DL><DT><DD><TT><PRE>"¿¿¿¿¿ ¿¿"L"¿¿¿¿¿ ¿¿",</PRE></TT></DL>the former is an array of<TT>chars</TT>with 22 elementsand a null byte,while the latter is an array of<TT>unsigned</TT><TT>shorts</TT>(<TT>Runes</TT>)with 8 elements and a null<TT>Rune</TT>.</P><P>The Plan 9 library provides an output conversion function,<TT>print</TT>(analogous to<TT>printf</TT>),with formats<TT>%c</TT>,<TT>%C</TT>,<TT>%s</TT>,and<TT>%S</TT>.Since<TT>print</TT>produces text, its output is always UTF.The character conversion<TT>%c</TT>(lower case) masks its argumentto 8 bits before converting to UTF.Thus<TT>L'ÿ'</TT>and<TT>'ÿ'</TT>printed under<TT>%c</TT>will be identical,but<TT>L'ALPHA'</TT>will print as the Unicodecharacter with decimal value 177.The character conversion<TT>%C</TT>(upper case) masks its argumentto 16 bits before converting to UTF.Thus<TT>L'ÿ'</TT>and<TT>L'ALPHA'</TT>will print correctly under<TT>%C</TT>,but<TT>'ÿ'</TT>will not.The conversion<TT>%s</TT>(lower case)expects a pointer to<TT>char</TT>and copies UTF sequences up to a null byte.The conversion<TT>%S</TT>(upper case) expects a pointer to<TT>Rune</TT>andperforms sequential<TT>%C</TT>conversions until a null<TT>Rune</TT>is encountered.</P><P>Another problem in format conversionis the definition of<TT>%10s</TT>:does the number refer to bytes or characters?We decided that such formats were mostoften used to align output columns andso made the number count characters.Some programs, however, use the countto place blank-padded stringsin fixed-sized arrays.These programs must be found and corrected.</P><P>Here is a complete example:<DL><DT><DD><TT><PRE>#include <u.h> char c[] = "¿¿¿¿¿ ¿¿";Rune s[] = L"¿¿¿¿¿ ¿¿"; main(void){ print("%d, %d\n", sizeof(c), sizeof(s)); print("%s\n", c); print("%S\n", s);}</PRE></TT></DL></P><P>This program prints<TT>23,</TT><TT>18</TT>and then two identical lines ofUTF text.In practice,<TT>%S</TT>and<TT>L"..."</TT>are rare in programs; one reason isthat most formatted I/O is done in unconverted UTF.</P><H4>Ramifications</H4><P>All programs in Plan 9 now read and write text as UTF, not ASCII.This change breaks two deep-rooted symmetries implicit in most C programs:</P><DL COMPACT><DT>1.<DD>A character is no longer a<TT>char</TT>.<DT>2.<DD>The internal representation (Rune) of a character now differs from itsexternal representation (UTF).</dl><P>In the sections that follow,we show how these issues were faced in the layers ofsystem software from the operating system up to the applications.The effects are wide-reaching and often surprising.</P><H4>Operating system</H4><P>Since UTF is the only format for text in Plan 9,the interface to the operating system had to be converted to UTF.Text strings cross the interface in several places:command arguments,file names,user names (people can log in using their native name),error messages,and miscellaneous minor places such as commands to the I/O system.Little change was required: null-terminated UTF stringsare equivalent to null-terminated ASCII strings for most purposesof the operating system.The library routines described in the next section made thatchange straightforward.</P>Lines 1–500 of 1324. Load more
<html> <title> data </title> <body BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#330088" ALINK="#FF0044"> <H1>Hello World <br> or <br> ¿ALPHA¿¿MU¿¿ALPHA ¿¿¿MUEPSILON <br> or <br> ¿¿¿¿¿ ¿¿ </H1> <DL><DD><I>Rob Pike<br> Ken Thompson<br> <br> <br> rob,ken@plan9.bell-labs.com<br> </I></DL> <DL><DD><H4>ABSTRACT</H4> <DL> <DT><DT> <DD> NOTE:<I> Originally appeared, in a slightly different form, in Proc. of the Winter 1993 USENIX Conf., pp. 43-50, San Diego </I><DT> <DD></dl> <br> Plan 9 from Bell Labs has recently been converted from ASCII to an ASCII-compatible variant of the Unicode Standard, a 16-bit character set. In this paper we explain the reasons for the change, describe the character set and representation we chose, and present the programming models and software changes that support the new text format. Although we stopped short of full internationalization­for example, system error messages are in Unixese, not Japanese­we believe Plan 9 is the first system to treat the representation of all major languages on a uniform, equal footing throughout all its software. </DL> <H4>Introduction </H4> <P> The world is multilingual but most computer systems are based on English and ASCII. The first release of Plan 9 [Pike90], a new distributed operating system from Bell Laboratories, seemed a good occasion to correct this chauvinism. It is easier to make such deep changes when building new systems than by refitting old ones. </P> <P> The ANSI C standard [ANSIC] contains some guidance on the matter of `wide' and `multi-byte' characters but falls far short of solving the myriad associated problems. We could find no literature on how to convert a <I>system</I> to larger character sets, although some individual <I>programs</I> had been converted. This paper reports what we discovered as we explored the problem of representing multilingual text at all levels of an operating system, from the file system and kernel through the applications and up to the window system and display. </P> <P> Plan 9 has not been `internationalized': its manuals are in English, its error messages are in English, and it can display text that goes from left to right only. But before we can address these other problems, we need to handle, uniformly and comfortably, the textual representation of all the major written languages. That subproblem is richer than we had anticipated. </P> <H4>Standards </H4> <P> Our first step was to select a standard. At the time (January 1992), there were only two viable options: ISO 10646 [ISO10646] and Unicode [Unicode]. The documents describing both proposals were still in the draft stage. </P> <P> The draft of ISO 10646 was not very attractive to us. It defined a sparse set of 32-bit characters, which would be hard to implement and have punitive storage requirements. Also, the draft attempted to mollify national interests by allocating 16-bit subspaces to national committees to partition individually. The suggested mode of use was to ``flip'' between separate national standards to implement the international standard. This did not strike us as a sound basis for a character set. As well, transmitting 32-bit values in a byte stream, such as in pipes, would be expensive and hard to implement. Since the standard does not define a byte order for such transmission, the byte stream would also have to carry state to enable the values to be recovered. </P> <P> The Unicode Standard is a proposal by a consortium of mostly American computer companies formed to protest the technical failings of ISO 10646. It defines a uniform 16-bit code based on the principle of unification: two characters are the same if they look the same even though they are from different languages. This principle, called Han unification, allows the large Japanese, Chinese, and Korean character sets to be packed comfortably into a 16-bit representation. </P> <P> We chose the Unicode Standard for its technical merits and because its code space was better defined. Moreover, the Unicode Consortium was derailing the ISO 10646 standard. (Now, in 1995, ISO 10646 is a standard with one 16-bit group defined, which is almost exactly the Unicode Standard. As most people expected, the two standards bodies reached a détente and ISO 10646 and Unicode represent the same character set.) </P> <P> The Unicode Standard defines an adequate character set but an unreasonable representation. It states that all characters are 16 bits wide and are communicated and stored in 16-bit units. It also reserves a pair of characters (hexadecimal FFFE and FEFF) to detect byte order in transmitted text, requiring state in the byte stream. (The Unicode Consortium was thinking of files, not pipes.) To adopt this encoding, we would have had to convert all text going into and out of Plan 9 between ASCII and Unicode, which cannot be done. Within a single program, in command of all its input and output, it is possible to define characters as 16-bit quantities; in the context of a networked system with hundreds of applications on diverse machines by different manufacturers, it is impossible. </P> <P> We needed a way to adapt the Unicode Standard to the tools-and-pipes model of text processing embodied by the Unix system. To do that, we needed an ASCII-compatible textual representation of Unicode characters for transmission and storage. In the draft ISO standard there was an informative (non-required) Annex called UTF that provided a byte stream encoding of the 32-bit ISO code. The encoding uses multibyte sequences composed from the 190 printable characters of Latin-1 to represent character values larger than 159. </P> <P> The UTF encoding has several good properties. By far the most important is that a byte in the ASCII range 0-127 represents itself in UTF. Thus UTF is backward compatible with ASCII. </P> <P> UTF has other advantages. It is a byte encoding and is therefore byte-order independent. ASCII control characters appear in the byte stream only as themselves, never as an element of a sequence encoding another character, so newline bytes separate lines of UTF text. Finally, ANSI C's <TT>strcmp</TT> function applied to UTF strings preserves the ordering of Unicode characters. </P> <P> To encode and decode UTF is expensive (involving multiplication, division, and modulo operations) but workable. UTF's major disadvantage is that the encoding is not self-synchronizing. It is in general impossible to find the character boundaries in a UTF string without reading from the beginning of the string, although in practice control characters such as newlines, tabs, and blanks provide synchronization points. </P> <P> In August 1992, X-Open circulated a proposal for another UTF-like byte encoding of Unicode characters. Their major concern was that an embedded character in a file name (in particular a slash) could be part of an escape sequence in UTF and therefore confuse a traditional file system. Their proposal would allow all 7-bit ASCII characters to represent themselves <I>and only themselves</I> in text. Multibyte sequences would contain only characters with the high bit set. We proposed a modification to the new UTF that would address our synchronization problem. Our proposal, which was originally known informally as UTF-2 and FSS-UTF, is now referred to as UTF-8 and has been approved by ISO to become Annex P to ISO 10646. </P> <P> The model for text in Plan 9 is chosen from these three standards*: </P> <DL> <DT><DT> <DD> NOTE:<I> * ``That's the nice thing about standards­there's so many to choose from.'' - Andy Tannenbaum (no, the other one) </I><DT> <DD></dl> <br> the Unicode character set encoded as a byte stream by UTF-8, from (soon to be) Annex P of ISO 10646. Although this mixture may seem like a precarious position for us to adopt, it is not as bad as it sounds. ISO 10646 and the Unicode Standard have converged, other systems such as Linux have adopted the same character set and encoding, and the general feeling seems to be that Unicode and UTF-8 will be accepted as the way to exchange text between systems. The prognosis for wide acceptance is good. <P> There are a couple of aspects of the Unicode Standard we have not faced. One is the issue of right-to-left text such as Hebrew or Arabic. Since that is an issue of display, not representation, we believe we can defer that problem for the moment without affecting our ability to solve it later. Another issue is diacriticals and `combining characters', which cause overstriking of multiple Unicode characters. Although necessary for some scripts, such as Thai, Arabic, and Hebrew, such characters confuse the issues for Latin languages because they generate multiple representations for accented characters. ISO 10646 describes three levels of implementation; in Plan 9 we decided not to address the issue. Again, this can be labeled as a display issue and its finer points are still being debated, so we felt comfortable deferring. Mañana. </P> <P> Although we converted Plan 9 in the altruistic interests of serving foreign languages, we have found the large character set attractive for other reasons. The Unicode Standard includes many characters­mathematical symbols, scientific notation, more general punctuation, and more­that we now use daily in our work. We no longer test our imaginations to find ways to include non-ASCII symbols in our text; why type <TT>:-)</TT> when you can use the character ¿? Most compelling is the ability to absorb documents and data that contain non-ASCII characters; our browser for the Oxford English Dictionary lets us see the dictionary as it really is, with pronunciation in the IPA font, foreign phrases properly rendered, and so on, <I>in plain text.</I> </P> <P> In the rest of this paper, except when stated otherwise, the term `UTF' refers to the UTF-8 encoding of Unicode characters as adopted by Plan 9. </P> <H4>C Compiler </H4> <P> The first program to be converted to UTF was the C Compiler. There are two levels of conversion. On the syntactic level, input to the C compiler is UTF; on the semantic level, the C language needs to define how compiled programs manipulate the UTF set. </P> <P> The syntactic part is simple. The ANSI C language standard defines the source character set to be ASCII. Since UTF is backward compatible with ASCII, the compiler needs little change. The only places where a larger character set is allowed are in character constants, strings, and comments. Since 7-bit ASCII characters can represent only themselves in UTF, the compiler does not have to be careful while looking for the termination of a string or comment. </P> <P> The Plan 9 compiler extends ANSI C to treat any Unicode character with a value outside of the ASCII range as an alphabetic. To a Greek programmer or an English mathematician, ALPHA is a sensible and now valid variable name. </P> <P> On the semantic level, ANSI C allows, but does not tie down, the notion of a <I>wide character</I> and admits string and character constants of this type. We chose the wide character type to be <TT>unsigned</TT> <TT>short</TT>. In the libraries, the word <TT>Rune</TT> is defined by a <TT>typedef</TT> to be equivalent to <TT>unsigned</TT> <TT>short</TT> and is used to signify a Unicode character. </P> <P> There are surprises; for example: <DL><DT><DD><TT><PRE> L'x' is 120 'x' is 120 L'ÿ' is 255 'ÿ' is -1, stdio EOF (if char is signed) L'ALPHA' is 945 'ALPHA' is illegal </PRE></TT></DL> In the string constants, <DL><DT><DD><TT><PRE> "¿¿¿¿¿ ¿¿" L"¿¿¿¿¿ ¿¿", </PRE></TT></DL> the former is an array of <TT>chars</TT> with 22 elements and a null byte, while the latter is an array of <TT>unsigned</TT> <TT>shorts</TT> (<TT>Runes</TT>) with 8 elements and a null <TT>Rune</TT>. </P> <P> The Plan 9 library provides an output conversion function, <TT>print</TT> (analogous to <TT>printf</TT>), with formats <TT>%c</TT>, <TT>%C</TT>, <TT>%s</TT>, and <TT>%S</TT>. Since <TT>print</TT> produces text, its output is always UTF. The character conversion <TT>%c</TT> (lower case) masks its argument to 8 bits before converting to UTF. Thus <TT>L'ÿ'</TT> and <TT>'ÿ'</TT> printed under <TT>%c</TT> will be identical, but <TT>L'ALPHA'</TT> will print as the Unicode character with decimal value 177. The character conversion <TT>%C</TT> (upper case) masks its argument to 16 bits before converting to UTF. Thus <TT>L'ÿ'</TT> and <TT>L'ALPHA'</TT> will print correctly under <TT>%C</TT>, but <TT>'ÿ'</TT> will not. The conversion <TT>%s</TT> (lower case) expects a pointer to <TT>char</TT> and copies UTF sequences up to a null byte. The conversion <TT>%S</TT> (upper case) expects a pointer to <TT>Rune</TT> and performs sequential <TT>%C</TT> conversions until a null <TT>Rune</TT> is encountered. </P> <P> Another problem in format conversion is the definition of <TT>%10s</TT>: does the number refer to bytes or characters? We decided that such formats were most often used to align output columns and so made the number count characters. Some programs, however, use the count to place blank-padded strings in fixed-sized arrays. These programs must be found and corrected. </P> <P> Here is a complete example: <DL><DT><DD><TT><PRE> #include <u.h> char c[] = "¿¿¿¿¿ ¿¿"; Rune s[] = L"¿¿¿¿¿ ¿¿"; main(void) { print("%d, %d\n", sizeof(c), sizeof(s)); print("%s\n", c); print("%S\n", s); } </PRE></TT></DL> </P> <P> This program prints <TT>23,</TT> <TT>18</TT> and then two identical lines of UTF text. In practice, <TT>%S</TT> and <TT>L"..."</TT> are rare in programs; one reason is that most formatted I/O is done in unconverted UTF. </P> <H4>Ramifications </H4> <P> All programs in Plan 9 now read and write text as UTF, not ASCII. This change breaks two deep-rooted symmetries implicit in most C programs: </P> <DL COMPACT> <DT>1.<DD> A character is no longer a <TT>char</TT>. <DT>2.<DD> The internal representation (Rune) of a character now differs from its external representation (UTF). </dl> <P> In the sections that follow, we show how these issues were faced in the layers of system software from the operating system up to the applications. The effects are wide-reaching and often surprising. </P> <H4>Operating system </H4> <P> Since UTF is the only format for text in Plan 9, the interface to the operating system had to be converted to UTF. Text strings cross the interface in several places: command arguments, file names, user names (people can log in using their native name), error messages, and miscellaneous minor places such as commands to the I/O system. Little change was required: null-terminated UTF strings are equivalent to null-terminated ASCII strings for most purposes of the operating system. The library routines described in the next section made that change straightforward. </P>