Received: from mail.proteosys.com ([62.225.9.49]) by nummer-3.proteosys with Microsoft SMTPSVC(5.0.2195.5329); Fri, 17 Jan 2003 20:41:20 +0100 Received: by mail.proteosys.com (8.12.2/8.12.2) with ESMTP id h0HJfH6C024286 for ; Fri, 17 Jan 2003 20:41:18 +0100 Received: from listserv.uni-heidelberg.de (listserv.uni-heidelberg.de [129.206.100.27]) by relay.uni-heidelberg.de (8.12.4/8.12.4) with ESMTP id h0HJXxEV004712; Fri, 17 Jan 2003 20:33:59 +0100 (MET) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C2BE60.688CF000" Received: from listserv (listserv.uni-heidelberg.de [129.206.100.27]) by listserv.uni-heidelberg.de (8.12.2/8.12.2/SuSE Linux 0.6) with ESMTP id h0H4jk3p014095; Fri, 17 Jan 2003 20:26:30 +0100 Received: from LISTSERV.UNI-HEIDELBERG.DE by LISTSERV.UNI-HEIDELBERG.DE (LISTSERV-TCP/IP release 1.8d) with spool id 7433 for LATEX-L@LISTSERV.UNI-HEIDELBERG.DE; Fri, 17 Jan 2003 20:26:30 +0100 X-MimeOLE: Produced By Microsoft Exchange V6.5 Received: from relay.uni-heidelberg.de (relay.uni-heidelberg.de [129.206.100.212]) by listserv.uni-heidelberg.de (8.12.2/8.12.2/SuSE Linux 0.6) with ESMTP id h0HJQUkr021011 for ; Fri, 17 Jan 2003 20:26:30 +0100 Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.189]) by relay.uni-heidelberg.de (8.12.4/8.12.4) with ESMTP id h0HJXQEV004680 for ; Fri, 17 Jan 2003 20:33:27 +0100 (MET) Received: from [212.227.126.155] (helo=mrelayng.kundenserver.de) by moutng.kundenserver.de with esmtp (Exim 3.35 #1) id 18ZcF4-0006rz-00 for LATEX-L@listserv.uni-heidelberg.de; Fri, 17 Jan 2003 20:33:26 +0100 Received: from [80.129.6.107] (helo=istrati.mittelbach-online.de) by mrelayng.kundenserver.de with asmtp (Exim 3.35 #1) id 18ZcF3-00011E-00 for LATEX-L@listserv.uni-heidelberg.de; Fri, 17 Jan 2003 20:33:26 +0100 Received: (from frank@localhost) by istrati.mittelbach-online.de (8.11.2/8.11.2/SuSE Linux 8.11.1-0.5) id h0HJVGg30406; Fri, 17 Jan 2003 20:31:16 +0100 In-Reply-To: <20030116114637.GA9844@g113.hadiko.de> References: <200212031601.gB3G11cQ009558@sun.dante.de> <15899.14827.804209.458595@istrati.mittelbach-online.de> <20030116114637.GA9844@g113.hadiko.de> Return-Path: X-Mailer: VM 6.96 under Emacs 20.7.1 X-OriginalArrivalTime: 17 Jan 2003 19:41:20.0080 (UTC) FILETIME=[68992500:01C2BE60] x-mime-autoconverted: from quoted-printable to 8bit by listserv.uni-heidelberg.de id h0HJQUkr021012 X-Authentication-Warning: istrati.mittelbach-online.de: frank set sender to frank@mittelbach-online.de using -f X-Scanned-By: MIMEDefang 2.28 (www . roaringpenguin . com / mimedefang) X-Spam-Score: -2.3 () EMAIL_ATTRIBUTION,IN_REP_TO,REFERENCES,SPAM_PHRASE_00_01,X_AUTH_WARNING Content-class: urn:content-classes:message Subject: Re: latex/3480: Support for UTF-8 missing in inputenc.sty Date: Fri, 17 Jan 2003 20:31:16 +0100 Message-ID: A<15912.23044.419984.897093@istrati.mittelbach-online.de> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Re: latex/3480: Support for UTF-8 missing in inputenc.sty Thread-Index: AcK+YGiz0iCPihLSRKWmp1S2HiIuLQ== From: "Frank Mittelbach" To: Reply-To: "Mailing list for the LaTeX3 project" Status: R X-Status: X-Keywords: X-UID: 4433 This is a multi-part message in MIME format. ------_=_NextPart_001_01C2BE60.688CF000 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dominique wrote: > I want to add several comments to Frank and Chris's utf8.def: good > =3D=3D=3D 1. The definition of the .dfu files. > > In the present model, we have the problem, that the same Unicode > character is defined several times in several .dfu files. If all > definitions are identical, this is no problem, but this has to be > ensured. Take the following example: Fontencoding LGR has the command > \euro, to be assigned to U+20AC, while TS1 has \texteuro, same = Unicode > character. if LGR does that then LGR is at fault since \texteuro is the LaTeX = internal character representation (LICR) name for the euro character [aside, where is that file, the one that i have here is very short and = doesn't contain \euro but neither looks like a proper encoding file either] i agree that there is a potential problem here, sort of similar to the potential problem that to inputenc files map the same abstract input to = a different internal command (of which only one should be a proper LICR) being pragmatic i believe that these get weed out after a while, the = reason for suggesting a .dfu file approach is that this allows easy extensions = for locally developed encodings. > Therefore I propose the following policy: > > - Unicode to TeX mappings are done in a single, fontencoding > independent file, e.g. ucs.map: > [...] > 0x20AC \texteuro > [...] > > - Fontencoding specific files contain list of supported code > positions, e.g. lgr.ucr and ts1.ucr (UCR=3D Unicode Range) both = contain > the number 0x20AC (but no more information). > > - A script then generates the .dfu files, the above example induces > the inclusion of > > \DeclareUnicodeCharacter{20AC}{\texteuro} > > into ts1.dfu and lgr.dfu (LGR has then to be updated to include the > macro \texteuro additionally to \euro). Note that only the final .dfu > files are seen by the latex executable, so this system does not > involve any changes in utf8.def. > > - The ucs.map file is managed by the LaTeX team. The .ucr files can = be > created be the developers of the fontencodings, thus enabling the > developement of fontencodings without the need of interaction with = the > LaTeX team. Inclusion of new into the ucs.map file should not be > subject to some restrictive election, since no resources are wasted, > unless some fontencoding requests these characters. in principle i agree with this kind of approach. however, i don't think = it is a very good idea to require a "script" (that then doesn't work on all installations or is not available on all installations ...) essentially per encoding Xenc.def there will only be the need to produce Xenc.dfu once (except for fixing it) and so people will distribute def = and dfu together anyway rather than distributing .def and .ucr and relying on = some process at the installation to generate .dfu for them. so i don't think this will work. i would suggest something simpler: a ucs.map file that contains the mappings Unicode->LICR in the form = directly usable in .dfu files, simply as a template for making a .dfu if really necessary. perhaps using docstrip to generate the standard dfu files from that file [further ideas welcome] > =3D=3D=3D 2. \IeC > > Most characters must be enclosed in a call to \IeC, like it is also > done by \DeclareInputText. Otherwise the following fragment > > \tableofcontents > \section{La=DF nach} % La\ss nach > > will give a TOC entry "La=DFnach" (i.e. the space will go away). criminal oversight. that certainly needs correction > =3D=3D=3D 3. Unicode to LaTeX mappings. > > There are already extensive lists of character mappings available at: > http://www.unruh.de/DniQ/latex/unicode/content/config/ so there is, worth stealing from > =3D=3D=3D 4. The loading of the .dfu files. > > It has been mentioned, that the late loading of the .dfu files (lines > 113--124) causes problems with saveboxes. For completeness I'd like = to > add, that also \xdef's etc. cause similar problems when used in the > preamble. > there is no such thing a \xdef (on free input) in LaTeX it should always = be \protected@edef or the like. having said that it doesn't really help as = the utf parsing expands straight up to the LICR (and that is not yet defined = at this point) so yes. you can formulate it differently: with the current = implementation this supports utf8 _after_ begin document with the outlined implmentation however that problem is going to vanish > =3D=3D=3D 5. Interoperability with ucs.sty > > There are some name clashes with my Unicode package. i'm not totally ignorant of your work, but it is a whileago that i = looked at it in some more detail and ... my impression was that it tries to provide much more than what we have = been after in the excerise for inputenc. would it be possible for you to give use a ten line bullet list of = comparsion? perhaps the best is simply to forget about what we did on lazy = afternoons during the Xmas holidays? > - utf8.def: I accept the fact, that this is the canonical name for > that file and will rename my inputencoding in favour of the kernel's > encoding. negotiable certainly. you try to hook into inputenc as well aren't you? > - \DeclareUnicodeCharacter: This command is named identically in my > system. I would appreciate if another name could be chosen at this > early stadium to evade chaos. what are your arguments? > Some possible names would be > > \DeclareUnicodeGlyph (according to the nomenclature of the Unicode > standard) no we map to LICR those are characters not glyphs! > \DeclareUnicodeCommand (analogous to \DeclareTextCommand) no again Command in that context has already some semantics maybe \DeclareUnicodeLaTeXMapping frank ------_=_NextPart_001_01C2BE60.688CF000 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: latex/3480: Support for UTF-8 missing in = inputenc.sty

Dominique wrote:

 > I want to add several comments to Frank and = Chris's utf8.def:

good

 > =3D=3D=3D 1. The definition of the .dfu = files.
 >
 > In the present model, we have the problem, = that the same Unicode
 > character is defined several times in = several .dfu files. If all
 > definitions are identical, this is no = problem, but this has to be
 > ensured. Take the following example: = Fontencoding LGR has the command
 > \euro, to be assigned to U+20AC, while TS1 = has \texteuro, same Unicode
 > character.

if LGR does that then LGR is at fault since \texteuro = is the LaTeX internal
character representation (LICR) name for the euro = character

[aside, where is that file, the one that i have here = is very short and doesn't
contain \euro but neither looks like a proper = encoding file either]

i agree that there is a potential problem here, sort = of similar to the
potential problem that to inputenc files map  = the same abstract input to a
different internal command (of which only one should = be a proper LICR)

being pragmatic i believe that these get weed out = after a while, the reason
for suggesting a .dfu file approach is that this = allows easy extensions for
locally developed encodings.

 > Therefore I propose the following = policy:
 >
 > - Unicode to TeX mappings are done in a = single, fontencoding
 > independent file, e.g. ucs.map:
 > [...]
 > 0x20AC   \texteuro
 > [...]
 >
 > - Fontencoding specific files contain list = of supported code
 > positions, e.g.  lgr.ucr and ts1.ucr = (UCR=3D Unicode Range) both contain
 > the number 0x20AC (but no more = information).
 >
 > - A script then generates the .dfu files, = the above example induces
 > the inclusion of
 >
 > = \DeclareUnicodeCharacter{20AC}{\texteuro}
 >
 > into ts1.dfu and lgr.dfu (LGR has then to = be updated to include the
 > macro \texteuro additionally to \euro). = Note that only the final .dfu
 > files are seen by the latex executable, so = this system does not
 > involve any changes in utf8.def.
 >
 > - The ucs.map file is managed by the LaTeX = team. The .ucr files can be
 > created be the developers of the = fontencodings, thus enabling the
 > developement of fontencodings without the = need of interaction with the
 > LaTeX team. Inclusion of new into the = ucs.map file should not be
 > subject to some restrictive election, = since no resources are wasted,
 > unless some fontencoding requests these = characters.

in principle i agree with this kind of approach. = however, i don't think it is
a very good idea to require a "script" = (that then doesn't work on all
installations or is not available on all = installations ...)

essentially per encoding Xenc.def there will only be = the need to produce
Xenc.dfu once (except for fixing it) and so people = will distribute def and dfu
together anyway rather than distributing .def and = .ucr and relying on some
process at the installation to generate .dfu for = them.

so i don't think this will work.

i would suggest something simpler:

a ucs.map file that contains the mappings = Unicode->LICR  in the form directly
usable in .dfu files, simply as a template for making = a .dfu if really
necessary.

perhaps using docstrip to generate the standard dfu = files from that file

[further ideas welcome]


 > =3D=3D=3D 2. \IeC
 >
 > Most characters must be enclosed in a call = to \IeC, like it is also
 > done by \DeclareInputText. Otherwise the = following fragment
 >
 > \tableofcontents
 > \section{La=DF nach}  % La\ss  = nach
 >
 > will give a TOC entry = "La=DFnach" (i.e. the space will go away).


criminal oversight. that certainly needs = correction



 > =3D=3D=3D 3. Unicode to LaTeX = mappings.
 >
 > There are already extensive lists of = character mappings available at:
 > http://ww= w.unruh.de/DniQ/latex/unicode/content/config/

so there is, worth stealing from



 > =3D=3D=3D 4. The loading of the .dfu = files.
 >
 > It has been mentioned, that the late = loading of the .dfu files (lines
 > 113--124) causes problems with saveboxes. = For completeness I'd like to
 > add, that also \xdef's etc. cause similar = problems when used in the
 > preamble.
 >

there is no such thing a \xdef (on free input) in = LaTeX it should always be
\protected@edef or the like. having said that it = doesn't really help as the
utf parsing expands straight up to the LICR (and that = is not yet defined at
this point)

so yes. you can formulate it differently: with the = current implementation this
supports utf8 _after_ begin document

with the outlined implmentation however that problem = is going to vanish




 > =3D=3D=3D 5. Interoperability with = ucs.sty
 >
 > There are some name clashes with my = Unicode package.

i'm not totally ignorant of your work, but it is a = whileago that i looked at
it in some more detail and ...

my impression was that it tries to provide much more = than what we have been
after in the excerise for inputenc.

would it be possible for you to give use a ten line = bullet list of comparsion?

perhaps the best is simply to forget about what we did = on lazy afternoons
during the Xmas holidays?


 > - utf8.def: I accept the fact, that this is = the canonical name for
 > that file and will rename my inputencoding = in favour of the kernel's
 > encoding.

negotiable certainly. you try to hook into inputenc as = well aren't you?

 > - \DeclareUnicodeCharacter: This command is = named identically in my
 > system. I would appreciate if another name = could be chosen at this
 > early stadium to evade chaos.

what are your arguments?


 >  Some possible names would be
 >
 > \DeclareUnicodeGlyph (according to the = nomenclature of the Unicode
 > standard)

no we map to LICR those are characters not = glyphs!

 > \DeclareUnicodeCommand (analogous to = \DeclareTextCommand)

no again Command in that context has already some = semantics

maybe

 \DeclareUnicodeLaTeXMapping


frank

------_=_NextPart_001_01C2BE60.688CF000--