code.delx.au - gnu-emacs/blob - admin/notes/unicode

   1                                             -*-mode: text; coding: utf-8;-*-
   2
   3 Copyright (C) 2002-2015 Free Software Foundation, Inc.
   4 See the end of the file for license conditions.
   5
   6 Importing a new Unicode Standard version into Emacs
   7 -------------------------------------------------------------
   8
   9 Emacs uses the following files from the Unicode Character Database
  10 (a.k.a. "UCD):
  11
  12   . UnicodeData.txt
  13   . BidiMirroring.txt
  14   . IVD_Sequences.txt
  15
  16 First, these files need to be copied into admin/unidata/, and then
  17 Emacs should be rebuilt for them to take effect.  Rebuilding Emacs
  18 updates several derived files elsewhere in the Emacs source tree,
  19 mainly in lisp/international/.
  20
  21 When Emacs is rebuilt for the first time after importing the new
  22 files, pay attention to any warning or error messages.  In particular,
  23 admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
  24 new bidirectional attributes of characters, because unidata-gen.el,
  25 bidi.c and dispextern.h need to be updated in that case; failure to do
  26 so will cause aborts in redisplay.
  27
  28 Next, review the changes in UnicodeData.txt vs the previous version
  29 used by Emacs.  Any changes, be it introduction of new scripts or
  30 addition of codepoints to existing scripts, might need corresponding
  31 changes in the data used for filling the category-table, case-table,
  32 and char-width-table.  The additional scripts should cause automatic
  33 updates in charscript.el, but it is a good idea to look at the results
  34 and see if any changes in admin/unidata/blocks.awk are required.
  35
  36 Any new scripts added by UnicodeData.txt will also need updates to
  37 script-representative-chars defined in fontset.el.  Other databases in
  38 fontset.el might also need to be updated as needed.
  39
  40 Problems, fixmes and other unicode-related issues
  41 -------------------------------------------------------------
  42
  43 Notes by fx to record various things of variable importance.  Handa
  44 needs to check them -- don't take too seriously, especially with
  45 regard to completeness.
  46
  47  * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
  48    undesirable effects.  E.g.:
  49    (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
  50    (multibyte-string-p (concat [?£])) => nil
  51    (text-char-description ?£) => "M-#"
  52
  53         These examples are all fixed by the change of 2002-10-14, but
  54         there still exist questionable SINGLE_BYTE_CHAR_P in the
  55         code (keymap.c and print.c).
  56
  57  * Rationalize character syntax and its relationship to the Unicode
  58    database.  (Applies mainly to symbol an punctuation syntax.)
  59
  60  * Fontset handling and customization needs work.  We want to relate
  61    fonts to scripts, probably based on the Unicode blocks.  The
  62    presence of small-repertoire 10646-encoded fonts in XFree 4 is a
  63    pain, not currently worked round.
  64
  65         With the change on 2002-07-26, multiple fonts can be
  66         specified in a fontset for a specific range of characters.
  67         Each range can also be specified by script.  Before using
  68         ISO10646 fonts, Emacs checks their repertories to avoid such
  69         fonts that don't have a glyph for a specific character.
  70
  71         fx has worked on fontset customization, but was stymied by
  72         basic problems with the way the default face is dealt with
  73         (and something else, I think).  This needs revisiting.
  74
  75  * Work is also needed on charset and coding system priorities.
  76
  77  * The relevant bits of latin1-disp.el need porting (and probably
  78    re-naming/updating).  See also cyril-util.el.
  79
  80  * Quail files need more work now the encoding is largely irrelevant.
  81
  82  * What to do with the old coding categories stuff?
  83
  84  * The preferred-coding-system property of charsets should probably be
  85    junked unless it can be made more useful now.
  86
  87  * find-multibyte-characters needs looking at.
  88
  89  * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
  90    charsets.
  91
  92  * Lazy-load tables for unify-charset somehow?
  93
  94         Actually, Emacs clears out all charset maps and unify-map just
  95         before dumping, and they are loaded again on demand by the
  96         dumped emacs.  But, those maps (char tables) generated while
  97         temacs is running can't be removed from the dumped emacs.
  98
  99  * iso-2022 charsets get unified on i/o.
 100
 101         With the change on 2003-01-06, decoding routines put the 'charset'
 102         property onto decoded text, and iso-2022 encoder pay attention
 103         to it.  Thus, for instance, reading and writing by
 104         iso-2022-7bit preserve the original designation sequences.
 105         The property name 'preferred-charset' may be better?
 106
 107         We may have to utilize this property to decide a font.
 108
 109  * Revisit locale processing: look at treating the language and
 110    charset parts separately.  (Language should affect things like
 111    spelling and calendar, but that's not a Unicode issue.)
 112
 113  * Handle Unicode combining characters usefully, e.g. diacritics, and
 114    handle more scripts specifically (à la Devanagari).  There are
 115    issues with canonicalization.
 116
 117  * We need tabular input methods, e.g. for maths symbols.  (Not
 118    specific to Unicode.)
 119
 120  * Need multibyte text in menus, e.g. for the above.  (Not specific to
 121    Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
 122
 123  * There's currently no support for Unicode normalization.
 124
 125  * Populate char-width-table correctly for Unicode characters and
 126    worry about what happens when double-width charsets covering
 127    non-CJK characters are unified.
 128
 129  * There are type errors lurking, e.g. in
 130    Fcheck_coding_systems_region.  Define ENABLE_CHECKING to find them.
 131
 132  * Old auto-save files, and similar files, such as Gnus drafts,
 133    containing non-ASCII characters probably won't be re-read correctly.
 134
 135
 136 Source file encoding
 137 --------------------
 138
 139 Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
 140 subset), but there are a few exceptions, listed below.  Perhaps
 141 someday many of these files will be converted to UTF-8, for
 142 convenience when using tools like 'grep -r', but this might need
 143 nontrivial changes to the build process.
 144
 145  * chinese-big5
 146
 147      These are verbatim copies of files taken from external sources.
 148      They haven't been converted to UTF-8.
 149
 150         leim/CXTERM-DIC/4Corner.tit
 151         leim/CXTERM-DIC/ARRAY30.tit
 152         leim/CXTERM-DIC/ECDICT.tit
 153         leim/CXTERM-DIC/ETZY.tit
 154         leim/CXTERM-DIC/PY-b5.tit
 155         leim/CXTERM-DIC/Punct-b5.tit
 156         leim/CXTERM-DIC/QJ-b5.tit
 157         leim/CXTERM-DIC/ZOZY.tit
 158         leim/MISC-DIC/CTLau-b5.html
 159         leim/MISC-DIC/cangjie-table.b5
 160
 161  * chinese-iso-8bit
 162
 163      These are verbatim copies of files taken from external sources.
 164      They haven't been converted to UTF-8.
 165
 166         leim/CXTERM-DIC/CCDOSPY.tit
 167         leim/CXTERM-DIC/Punct.tit
 168         leim/CXTERM-DIC/QJ.tit
 169         leim/CXTERM-DIC/SW.tit
 170         leim/CXTERM-DIC/TONEPY.tit
 171         leim/MISC-DIC/CTLau.html
 172         leim/MISC-DIC/pinyin.map
 173         leim/MISC-DIC/ziranma.cin
 174
 175  * cp850
 176
 177      This file contains non-ASCII characters in unibyte strings.  When
 178      editing a keyboard layout it's more convenient to see 'é' than
 179      '\202', and the MS-DOS compiler requires the single byte if a
 180      backslash escape is not being used.
 181
 182         src/msdos.c
 183
 184  * iso-2022-cn-ext
 185
 186      This file is externally generated from leim/MISC-DIC/cangjie-table.b5
 187      by Big5->CNS converter.  It hasn't been converted to UTF-8.
 188
 189         leim/MISC-DIC/cangjie-table.cns
 190
 191  * japanese-iso-8bit
 192
 193      SKK-JISYO.L is a verbatim copy of a file taken from an external source.
 194      It hasn't been converted to UTF-8.
 195
 196         leim/SKK-DIC/SKK-JISYO.L
 197
 198  * japanese-shift-jis
 199
 200      This is a verbatim copy of a file taken from an external source.
 201      It hasn't been converted to UTF-8.
 202
 203         admin/charsets/mapfiles/cns2ucsdkw.txt
 204
 205  * iso-2022-7bit
 206
 207      This file switches between CJK charsets, which is not encoded in UTF-8.
 208
 209         etc/HELLO
 210
 211      Each of these files contains just one CJK charset, but Emacs
 212      currently has no easy way to specify set-charset-priority on a
 213      per-file basis, so converting any of these files to UTF-8 might
 214      change the file's appearance when viewed by an Emacs that is
 215      operating in some other language environment.
 216
 217         etc/tutorials/TUTORIAL.ja
 218         lisp/international/ja-dic-cnv.el
 219         lisp/international/ja-dic-utl.el
 220         lisp/international/kinsoku.el
 221         lisp/international/kkc.el
 222         lisp/international/titdic-cnv.el
 223         lisp/language/japan-util.el
 224         lisp/language/japanese.el
 225         lisp/leim/quail/cyril-jis.el
 226         lisp/leim/quail/hanja-jis.el
 227         lisp/leim/quail/japanese.el
 228         lisp/leim/quail/py-punct.el
 229         lisp/leim/quail/pypunct-b5.el
 230
 231      This file contains just Chinese characters, and has same problem.
 232      Also, it contains characters that cannot be encoded in UTF-8.
 233
 234         lisp/international/titdic-cnv.el
 235
 236  * utf-8-emacs
 237
 238      These files contain characters that cannot be encoded in UTF-8.
 239
 240         lisp/language/ethio-util.el
 241         lisp/language/ethiopic.el
 242         lisp/language/ind-util.el
 243         lisp/language/tibet-util.el
 244         lisp/language/tibetan.el
 245         lisp/leim/quail/ethiopic.el
 246         lisp/leim/quail/tibetan.el
 247
 248  * binary files
 249
 250      These files contain binary data, and are not text files.
 251      Some of the entries in this list are patterns, and stand for any
 252      files with the listed extension.
 253
 254         *.gz
 255         *.icns
 256         *.ico
 257         *.pbm
 258         *.pdf
 259         *.png
 260         *.sig
 261         etc/e/eterm-color
 262         etc/package-keyring.gpg
 263         msdos/emacs.pif
 264         nextstep/GNUstep/Emacs.base/Resources/emacs.tiff
 265         nt/icons/hand.cur
 266
 267 \f
 268 This file is part of GNU Emacs.
 269
 270 GNU Emacs is free software: you can redistribute it and/or modify
 271 it under the terms of the GNU General Public License as published by
 272 the Free Software Foundation, either version 3 of the License, or
 273 (at your option) any later version.
 274
 275 GNU Emacs is distributed in the hope that it will be useful,
 276 but WITHOUT ANY WARRANTY; without even the implied warranty of
 277 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 278 GNU General Public License for more details.
 279
 280 You should have received a copy of the GNU General Public License
 281 along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.