henesyinit from p9f47575dc5 years agoHistory
1229 lines · 39.9 KB · Text
RawDownload
.tr -\(hy.TLHello World.bror.brΚαλημέρα κόσμε.bror.brこんにちは 世界.AURob PikeKen Thompson.AI.MH.ABPlan 9 from Bell Labs has recently been converted from ASCIIto an ASCII-compatible variant of Unicode, a 16-bit character set.In this paper we explain the reasons for the change,describe the character set and representation we chose,and present the programming models and software changesthat support the new text format.Although we stopped short of full internationalization\(emforexample, system error messages are in Unixese, not Japanese\(emwebelieve Plan 9 is the first system to treat the representationof all major languages on a uniform, equal footing throughout all itssoftware..AE.SHIntroduction.PPThe world is multilingual but most computer systemsare based on English and ASCII.The release of Plan 9 [Pike90], a new distributed operatingsystem from Bell Laboratories, seemed a good occasionto correct this chauvinism.It is easier to make such deep changes when building new systems thanby refitting old ones..PPThe ANSI C standard [ANSIC] contains some guidance on the matter of`wide' and `multi-byte' characters but falls far short ofsolving the myriad associated problems.We could find no literature on how to convert a.I systemto larger character sets, although some individual.I programshad been converted.This paper reports what we discovered as weexplored the problem of representing multilingualtext at all levels of an operating system,from the file system and kernel throughthe applications and up to the window systemand display..PPPlan 9 has not been `internationalized':its manuals are in English,its error messages are in English,and it can display text that goes from left to right only.But before we can address these other problems,we need to handle, uniformly and comfortably,the textual representation of all the major written languages.That subproblem is richer than we had anticipated..SHStandards.PPOur first step was to select a standard.At the time (January 1992),there were only two viable options:ISO 10646 [ISO10646] and Unicode [Unicode].The documents describing both proposals were still in the draft stage..PPISO 10646 was notvery attractive to us.The standard defines a sparse set of 32-bit characters,which would behard to implementand have punitive storage requirements.Also, the standard attempts tomollify national interests by allocating16-bit subspaces to national committeesto partition individually.The suggested mode of use is to``flip'' between separate nationalstandards to implement the international standard.This did not strike us as a sound basis for a character set.As well, transmitting 32-bit values in a byte stream,such as in pipes, would be expensive and hard to implement.Since the standard does not define a byte order for suchtransmission, the byte stream would also have to carrystate to enable the values to be recovered..PPUnicode is a proposal by a consortium of mostly Americancomputer companies formedto protest the technicalfailings of ISO 10646.Unicode defines a uniform 16-bit code based on theprinciple of unification:two characters are the same if they look thesame even though they are from differentlanguages.This principle, called Han unification,allows the large Japanese, Chinese, and Koreancharacter sets to be packed comfortably into a 16-bit representation..PPWe chose Unicode for its technical merits and because itscode space was better defined.Moreover,the existence of Unicode was derailing theISO 10646 standard.ISO 10646 is now in its second draft andhas only one 16-bit group defined,which is almost exactly Unicode.Most people expect the two standards bodiesto reach a détente so thatISO 10646 and Unicode will represent the same character set..PPUnicode defines an adequate character setbut an unreasonable representation.The Unicode standard states that all charactersare 16 bits wide and are communicated and stored in16-bit units.It also reserves a pair of characters(hexadecimal FFFE and FEFF) to detect byte orderin transmitted text, requiring state in the byte stream.(The Unicode committee was thinking of files, not pipes.)To adopt Unicode,we would have had to convert all text goinginto and out of Plan 9 between ASCII and Unicode, which cannot be done.Within a single program, in command of all its input and output,it is possible to define characters as 16-bit quantities;in the context of a networked system withhundreds of applications on diverse machinesby different manufacturers,it is impossible..PPWe needed a way to adapt Unicode to the tools-and-pipesmodel of text processing embodied by the Unix system.To do that, weneeded an ASCII-compatible textualrepresentation of Unicode for transmissionand storage.In the ISO standard there is an informative(non-required)Annexcalled UTFthat provides a byte stream encodingof the 32-bit ISO code.The encoding uses multibyte sequences composedfrom the 190 printable characters of Latin-1to represent character values largerthan 159..PPThe UTF encoding has several good properties.By far the most important is thata byte in the ASCII range 0-127 representsitself in UTF.Thus UTF is backward compatible with ASCII..PPUTF has other advantages.It is a byte encoding and istherefore byte-order independent.ASCII control characters appear in the byte streamonly as themselves, never as an element of a sequenceencoding another character,so newline bytes separate lines of UTF text.Finally, ANSI C's.CW strcmpfunction applied to UTF strings preserves the ordering of Unicode characters..PPTo encode and decode UTF is expensive (involving multiplication,division, and modulo operations) but workable.UTF's major disadvantage is that the encodingis not self-synchronizing.It is in general impossible to find the characterboundaries in a UTF string without reading fromthe beginning of the string, although in practicecontrol characters such as newlines,tabs, and blanks provide synchronization points..PPIn August 1992,X-Open circulated a proposal for another UTF-likebyte encoding of Unicode.Their major concern was that an embedded characterin a file name(in particular a slash)could be part of an escape sequence in UTF andtherefore confuse a traditional file system.Their proposal would allow all 7-bit ASCII charactersto represent themselves.I "and only themselves"in text.Multibyte sequences would contain only characterswith the high bit set.We proposed a modification to the new UTF thatwould address our synchronization problem.The modified new proposal is now informally called UTF-2and is being proposed as another informative Annex to ISO 10646..PPThe model for text in Plan 9 is chosen from thesethree standards*:.FS* ``That's the nice thing about standards\(emthere's so many to choose from.'' \- Andy Tannenbaum (no, the other one).FEthe Unicode character set encoded as byte stream byUTF-2, from an X-Open proposed modification ofAnnex F of ISO 10646.Although this may seem like a precarious position for us to adopt,it is not as bad as it sounds.If, as expected,ISO adopts Unicode as Group 0 of 10646and ISO publishes UTF-2 as an Annex,thenPlan 9 will be ISO/UTF-2 compatible..PPThere are a couple of aspects of Unicode we have not faced.One is the issue of right-to-left text such as Hebrew or Arabic.Since that is an issue of display, not representation, we believewe can defer that problem for the moment without affecting ourability to solve it later.Another issue is diacriticals, which cause overstriking of multipleUnicode characters.Again, these are display issues and, since the Unicodecommittee is still deciding their finer points,we felt comfortable deferring.  Mañana..PPAlthough we converted Plan 9 in the altruistic interests ofserving foreign languages, we have found the large characterset attractive for other reasons.  Unicode includes manycharacters\(emmathematical symbols, scientific notation,more general punctuation, and more\(emthat we now usedaily in our work.  We no longer test our imaginationsto find ways to include non-ASCII symbols in our text;why type.CW :-)when you can use the character ☺?Most compelling is the ability to absorb documentsand data that contain non-ASCII characters; our browser for theOxford English Dictionarylets us see the dictionary as it really is, with pronunciationin the IPA font, foreign phrases properly rendered, and so on,.I "in plain text..PPIn the rest of this paper, except whenstated otherwise, the term `UTF' refers to the UTF-2 encodingof Unicode characters as adopted by Plan 9..SHC Compiler.PPThe first program to be converted to UTFwas the C Compiler.There are two levels of conversion.On the syntactic level,input to the C compileris UTF; on the semantic level,the C language needs to definehow compiled programs manipulatethe UTF set..PPThe syntactic part is simple.The ANSI C language standard defines thesource character set to be ASCII.Since UTF is backward compatible with ASCII,the compiler needs little change.The only places where a larger character setis allowed are in character constants, strings, and comments.Since 7-bit ASCII characters can represent onlythemselves in UTF,the compiler does not have to be careful while lookingfor the termination of a string or comment..PPThe Plan 9 compiler extends ANSI C to treat any Unicodecharacter with a value outside of the ASCII range asan alphabetic.To a Greek programmer or an English mathematician,α is a sensible and now valid variable name..PPOn the semantic level, ANSI C allows,but does not tie down,the notion of a.I "wide characterand admits string and character constantsof this type.We chose the wide character type to be.CW unsigned.CW short .In the libraries, the word.CW Runeis defined by a.CW typedefto be equivalent to.CW unsigned.CW shortand isused to signify a Unicode character..PPThere are surprises; for example:.P1L'x'	\f1is 120\fP\&'x'	\f1is 120\fPL'ÿ'	\f1is 255\fP\&'ÿ'	\f1is \-1, stdio \fPEOF\f1 (if \fPchar\f1 is signed)\fPL'α'	\f1is 945\fP\&'α'	\f1is illegal\fP.P2In the string constants,.P1"こんにちは 世界"L"こんにちは 世界",.P2the former is an array of.CW charswith 22 elementsand a null byte,while the latter is an array of.CW unsigned.CW shorts.CW Runes ) (with 8 elements and a null.CW Rune ..PPThe Plan 9 library provides an output conversion function,.CW print(analogous to.CW printf ),with formats.CW %c ,.CW %C ,.CW %s ,and.CW %S .Since.CW printproduces text, its output is always UTF.The character conversion.CW %c(lower case) masks its argumentto 8 bits before converting to UTF.Thus.CW L'ÿ'and.CW 'ÿ'printed under.CW %cwill be identical,but.CW L'α'will print as the Unicodecharacter with decimal value 177.The character conversion.CW %C(upper case) masks its argumentto 16 bits before converting to UTF.Thus.CW L'ÿ'and.CW L'α'will print correctly under.CW %C ,but.CW 'ÿ'will not.The conversion.CW %s(lower case)expects a pointer to.CW charand copies UTF sequences up to a null byte.The conversion.CW %S(upper case) expects a pointer to.CW Runeandperforms sequential.CW %Cconversions until a null.CW Runeis encountered..PPAnother problem in format conversionis the definition of.CW %10s :does the number refer to bytes or characters?We decided that such formats were mostoften used to align output columns andso made the number count characters.Some programs, however, use the countto place blank-padded stringsin fixed-sized arrays.These programs must be found and corrected..PPHere is a complete example:.P1#include <u.h> char c[] = "こんにちは 世界";Rune s[] = L"こんにちは 世界"; main(void){	print("%d, %d\en", sizeof(c), sizeof(s));	print("%s\en", c);	print("%S\en", s);}.P2.PPThis program prints.CW 23,.CW 18and then two identical lines ofUTF text.In practice,.CW %Sand.CW L"..."are rare in programs; one reason isthat most formatted I/O is done in unconverted UTF..SHRamifications.PPAll programs in Plan 9 now read and write text as UTF, not ASCII.This change breaks two deep-rooted symmetries implicit in most C programs:.IP 1.A character is no longer a.CW char ..IP 2.The internal representation (Unicode) of a character now differs from itsexternal representation (UTF)..PPIn the sections that follow,we show how these issues were faced in the layers ofsystem software from the operating system up to the applications.The effects are wide-reaching and often surprising..SHOperating system.PPSince UTF is the only format for text in Plan 9,the interface to the operating system had to be converted to UTF.Text strings cross the interface in several places:command arguments,file names,user names (people can log in using their native name),error messages,and miscellaneous minor places such as commands to the I/O system.Little change was required: null-terminated UTF stringsare equivalent to null-terminated ASCII strings for most purposesof the operating system.The library routines described in the next section made thatchange straightforward..PPThe window system, once called.CW 8.5 ,is now rightfully called.CW 8½ ..SHLibraries.PPA header file included by all programs (see [Pike92]) declaresthe.CW Runetype to hold 16-bit character values:.P1typedef unsigned short Rune;.P2Also defined are several constants relevant to UTF:.P1enum{    UTFmax    = 3,    /* maximum bytes per rune */    Runesync  = 0x80, /* cannot represent part of a UTF sequence (<) */    Runeself  = 0x80, /* rune and UTF sequences are the same (<) */    Runeerror = 0x80, /* decoding error in UTF */};.P2(With the original UTF,.CW Runesyncwas hexadecimal 21 and.CW Runeselfwas A0.).CW UTFmaxbytes are sufficientto hold the UTF encoding of any Unicode character.Characters of value less than.CW Runesynconly appear in a UTF string asthemselves, never as part of a sequence encoding another character.Characters of value less than.CW Runeselfencode into single bytesof the same value.Finally, when the library detects errors in UTF input\(embyte sequencesthat are not valid UTF sequences\(emit converts the first byte of theerror sequence to the character.CW Runeerror .There is little a rune-oriented program can do when given bad dataexcept exit, which is unreasonable, or carry on.Originally the conversion routines, described below,returned errors when given invalid UTF,but we found ourselves repeatedly checking for errors and ignoring them.We therefore decided to convert a bad sequence to a valid rune

Lines 1–500 of 1229.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t