/* * Copyright (c) 2005, 2009, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License version 2 only, as * published by the Free Software Foundation.  Oracle designates this * particular file as subject to the "Classpath" exception as provided * by Oracle in the LICENSE file that accompanied this code. * * This code is distributed in the hope that it will be useful, but WITHOUT * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License * version 2 for more details (a copy is included in the LICENSE file that * accompanied this code). * * You should have received a copy of the GNU General Public License version * 2 along with this work; if not, write to the Free Software Foundation, * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. * * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA * or visit www.oracle.com if you need additional information or have any * questions. *//* ******************************************************************************* * (C) Copyright IBM Corp. and others, 1996-2009 - All Rights Reserved         * *                                                                             * * The original version of this source code and documentation is copyrighted   * * and owned by IBM, These materials are provided under terms of a License     * * Agreement between IBM and Sun. This technology is protected by multiple     * * US and International patents. This notice and attribution to IBM may not    * * to removed.                                                                 * ******************************************************************************* */ package sun.text.normalizer; import java.text.CharacterIterator;import java.text.Normalizer; /** * Unicode Normalization * * <h2>Unicode normalization API</h2> * * <code>normalize</code> transforms Unicode text into an equivalent composed or * decomposed form, allowing for easier sorting and searching of text. * <code>normalize</code> supports the standard normalization forms described in * <a href="http://www.unicode.org/unicode/reports/tr15/" target="unicode"> * Unicode Standard Annex #15 &mdash; Unicode Normalization Forms</a>. * * Characters with accents or other adornments can be encoded in * several different ways in Unicode.  For example, take the character A-acute. * In Unicode, this can be encoded as a single character (the * "composed" form): * * <p> *      00C1    LATIN CAPITAL LETTER A WITH ACUTE * </p> * * or as two separate characters (the "decomposed" form): * * <p> *      0041    LATIN CAPITAL LETTER A *      0301    COMBINING ACUTE ACCENT * </p> * * To a user of your program, however, both of these sequences should be * treated as the same "user-level" character "A with acute accent".  When you * are searching or comparing text, you must ensure that these two sequences are * treated equivalently.  In addition, you must handle characters with more than * one accent.  Sometimes the order of a character's combining accents is * significant, while in other cases accent sequences in different orders are * really equivalent. * * Similarly, the string "ffi" can be encoded as three separate letters: * * <p> *      0066    LATIN SMALL LETTER F *      0066    LATIN SMALL LETTER F *      0069    LATIN SMALL LETTER I * </p> * * or as the single character * * <p> *      FB03    LATIN SMALL LIGATURE FFI * </p> * * The ffi ligature is not a distinct semantic character, and strictly speaking * it shouldn't be in Unicode at all, but it was included for compatibility * with existing character sets that already provided it.  The Unicode standard * identifies such characters by giving them "compatibility" decompositions * into the corresponding semantic characters.  When sorting and searching, you * will often want to use these mappings. * * <code>normalize</code> helps solve these problems by transforming text into * the canonical composed and decomposed forms as shown in the first example * above. In addition, you can have it perform compatibility decompositions so * that you can treat compatibility characters the same as their equivalents. * Finally, <code>normalize</code> rearranges accents into the proper canonical * order, so that you do not have to worry about accent rearrangement on your * own. * * Form FCD, "Fast C or D", is also designed for collation. * It allows to work on strings that are not necessarily normalized * with an algorithm (like in collation) that works under "canonical closure", * i.e., it treats precomposed characters and their decomposed equivalents the * same. * * It is not a normalization form because it does not provide for uniqueness of * representation. Multiple strings may be canonically equivalent (their NFDs * are identical) and may all conform to FCD without being identical themselves. * * The form is defined such that the "raw decomposition", the recursive * canonical decomposition of each character, results in a string that is * canonically ordered. This means that precomposed characters are allowed for * as long as their decompositions do not need canonical reordering. * * Its advantage for a process like collation is that all NFD and most NFC texts * - and many unnormalized texts - already conform to FCD and do not need to be * normalized (NFD) for such a process. The FCD quick check will return YES for * most strings in practice. * * normalize(FCD) may be implemented with NFD. * * For more details on FCD see the collation design document: * http://source.icu-project.org/repos/icu/icuhtml/trunk/design/collation/ICU_collation_design.htm * * ICU collation performs either NFD or FCD normalization automatically if * normalization is turned on for the collator object. Beyond collation and * string search, normalized strings may be useful for string equivalence * comparisons, transliteration/transcription, unique representations, etc. * * The W3C generally recommends to exchange texts in NFC. * Note also that most legacy character encodings use only precomposed forms and * often do not encode any combining marks by themselves. For conversion to such * character encodings the Unicode text needs to be normalized to NFC. * For more usage examples, see the Unicode Standard Annex. * @stable ICU 2.8 */ public final class NormalizerBase implements Cloneable {     //-------------------------------------------------------------------------    // Private data    //-------------------------------------------------------------------------    private char[] buffer = new char[100];    private int bufferStart = 0;    private int bufferPos   = 0;    private int bufferLimit = 0;     // The input text and our position in it    private UCharacterIterator  text;    private Mode                mode = NFC;    private int                 options = 0;    private int                 currentIndex;    private int                 nextIndex;     /**     * Options bit set value to select Unicode 3.2 normalization     * (except NormalizationCorrections).     * At most one Unicode version can be selected at a time.     * @stable ICU 2.6     */    public static final int UNICODE_3_2=0x20;     /**     * Constant indicating that the end of the iteration has been reached.     * This is guaranteed to have the same value as {@link UCharacterIterator#DONE}.     * @stable ICU 2.8     */    public static final int DONE = UCharacterIterator.DONE;     /**     * Constants for normalization modes.     * @stable ICU 2.8     */    public static class Mode {        private int modeValue;        private Mode(int value) {            modeValue = value;        }         /**         * This method is used for method dispatch         * @stable ICU 2.6         */        protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                UnicodeSet nx) {            int srcLen = (srcLimit - srcStart);            int destLen = (destLimit - destStart);            if( srcLen > destLen ) {                return srcLen;            }            System.arraycopy(src,srcStart,dest,destStart,srcLen);            return srcLen;        }         /**         * This method is used for method dispatch         * @stable ICU 2.6         */        protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                int options) {            return normalize(   src, srcStart, srcLimit,                                dest,destStart,destLimit,                                NormalizerImpl.getNX(options)                                );        }         /**         * This method is used for method dispatch         * @stable ICU 2.6         */        protected String normalize(String src, int options) {            return src;        }         /**         * This method is used for method dispatch         * @stable ICU 2.8         */        protected int getMinC() {            return -1;        }         /**         * This method is used for method dispatch         * @stable ICU 2.8         */        protected int getMask() {            return -1;        }         /**         * This method is used for method dispatch         * @stable ICU 2.8         */        protected IsPrevBoundary getPrevBoundary() {            return null;        }         /**         * This method is used for method dispatch         * @stable ICU 2.8         */        protected IsNextBoundary getNextBoundary() {            return null;        }         /**         * This method is used for method dispatch         * @stable ICU 2.6         */        protected QuickCheckResult quickCheck(char[] src,int start, int limit,                                              boolean allowMaybe,UnicodeSet nx) {            if(allowMaybe) {                return MAYBE;            }            return NO;        }         /**         * This method is used for method dispatch         * @stable ICU 2.8         */        protected boolean isNFSkippable(int c) {            return true;        }    }     /**     * No decomposition/composition.     * @stable ICU 2.8     */    public static final Mode NONE = new Mode(1);     /**     * Canonical decomposition.     * @stable ICU 2.8     */    public static final Mode NFD = new NFDMode(2);     private static final class NFDMode extends Mode {        private NFDMode(int value) {            super(value);        }         protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                UnicodeSet nx) {            int[] trailCC = new int[1];            return NormalizerImpl.decompose(src,  srcStart,srcLimit,                                            dest, destStart,destLimit,                                            false, trailCC,nx);        }         protected String normalize( String src, int options) {            return decompose(src,false,options);        }         protected int getMinC() {            return NormalizerImpl.MIN_WITH_LEAD_CC;        }         protected IsPrevBoundary getPrevBoundary() {            return new IsPrevNFDSafe();        }         protected IsNextBoundary getNextBoundary() {            return new IsNextNFDSafe();        }         protected int getMask() {            return (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFD);        }         protected QuickCheckResult quickCheck(char[] src,int start,                                              int limit,boolean allowMaybe,                                              UnicodeSet nx) {            return NormalizerImpl.quickCheck(                                             src, start,limit,                                             NormalizerImpl.getFromIndexesArr(                                                                              NormalizerImpl.INDEX_MIN_NFD_NO_MAYBE                                                                              ),                                             NormalizerImpl.QC_NFD,                                             0,                                             allowMaybe,                                             nx                                             );        }         protected boolean isNFSkippable(int c) {            return NormalizerImpl.isNFSkippable(c,this,                                                (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFD)                                                );        }    }     /**     * Compatibility decomposition.     * @stable ICU 2.8     */    public static final Mode NFKD = new NFKDMode(3);     private static final class NFKDMode extends Mode {        private NFKDMode(int value) {            super(value);        }         protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                UnicodeSet nx) {            int[] trailCC = new int[1];            return NormalizerImpl.decompose(src,  srcStart,srcLimit,                                            dest, destStart,destLimit,                                            true, trailCC, nx);        }         protected String normalize( String src, int options) {            return decompose(src,true,options);        }         protected int getMinC() {            return NormalizerImpl.MIN_WITH_LEAD_CC;        }         protected IsPrevBoundary getPrevBoundary() {            return new IsPrevNFDSafe();        }         protected IsNextBoundary getNextBoundary() {            return new IsNextNFDSafe();        }         protected int getMask() {            return (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFKD);        }         protected QuickCheckResult quickCheck(char[] src,int start,                                              int limit,boolean allowMaybe,                                              UnicodeSet nx) {            return NormalizerImpl.quickCheck(                                             src,start,limit,                                             NormalizerImpl.getFromIndexesArr(                                                                              NormalizerImpl.INDEX_MIN_NFKD_NO_MAYBE                                                                              ),                                             NormalizerImpl.QC_NFKD,                                             NormalizerImpl.OPTIONS_COMPAT,                                             allowMaybe,                                             nx                                             );        }         protected boolean isNFSkippable(int c) {            return NormalizerImpl.isNFSkippable(c, this,                                                (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFKD)                                                );        }    }     /**     * Canonical decomposition followed by canonical composition.     * @stable ICU 2.8     */    public static final Mode NFC = new NFCMode(4);     private static final class NFCMode extends Mode{        private NFCMode(int value) {            super(value);        }        protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                UnicodeSet nx) {            return NormalizerImpl.compose( src, srcStart, srcLimit,                                           dest,destStart,destLimit,                                           0, nx);        }         protected String normalize( String src, int options) {            return compose(src, false, options);        }         protected int getMinC() {            return NormalizerImpl.getFromIndexesArr(                                                    NormalizerImpl.INDEX_MIN_NFC_NO_MAYBE                                                    );        }        protected IsPrevBoundary getPrevBoundary() {            return new IsPrevTrueStarter();        }        protected IsNextBoundary getNextBoundary() {            return new IsNextTrueStarter();        }        protected int getMask() {            return (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFC);        }        protected QuickCheckResult quickCheck(char[] src,int start,                                              int limit,boolean allowMaybe,                                              UnicodeSet nx) {            return NormalizerImpl.quickCheck(                                             src,start,limit,                                             NormalizerImpl.getFromIndexesArr(                                                                              NormalizerImpl.INDEX_MIN_NFC_NO_MAYBE                                                                              ),                                             NormalizerImpl.QC_NFC,                                             0,                                             allowMaybe,                                             nx                                             );        }        protected boolean isNFSkippable(int c) {            return NormalizerImpl.isNFSkippable(c,this,                                                ( NormalizerImpl.CC_MASK|NormalizerImpl.COMBINES_ANY|                                                  (NormalizerImpl.QC_NFC & NormalizerImpl.QC_ANY_NO)                                                  )                                                );        }    };     /**     * Compatibility decomposition followed by canonical composition.     * @stable ICU 2.8     */    public static final Mode NFKC =new NFKCMode(5);     private static final class NFKCMode extends Mode{        private NFKCMode(int value) {            super(value);        }        protected int normalize(char[] src, int srcStart, int srcLimit,                                char[] dest,int destStart,int destLimit,                                UnicodeSet nx) {            return NormalizerImpl.compose(src,  srcStart,srcLimit,                                          dest, destStart,destLimit,                                          NormalizerImpl.OPTIONS_COMPAT, nx);        }         protected String normalize( String src, int options) {            return compose(src, true, options);        }        protected int getMinC() {            return NormalizerImpl.getFromIndexesArr(                                                    NormalizerImpl.INDEX_MIN_NFKC_NO_MAYBE                                                    );        }        protected IsPrevBoundary getPrevBoundary() {            return new IsPrevTrueStarter();        }        protected IsNextBoundary getNextBoundary() {            return new IsNextTrueStarter();        }        protected int getMask() {            return (NormalizerImpl.CC_MASK|NormalizerImpl.QC_NFKC);        }        protected QuickCheckResult quickCheck(char[] src,int start,

Lines 1–500 of 1680. Load more