Content-Type: multipart/signed; boundary=Apple-Mail-6--966340416; micalg=sha1; protocol="application/pkcs7-signature"
Mime-Version: 1.0 (Apple Message framework v930.3)
Message-ID:  <11ECEE9E-C040-44DF-9D1F-97281D9128ED@gmail.com>
Date:         Mon, 16 Mar 2009 17:23:00 +1030
Reply-To: Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender: Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From: Will Robertson <wspr81@GMAIL.COM>
Subject: inputenc for XeTeX and LuaTeX
To: LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
Precedence: list
Status: R

--Apple-Mail-6--966340416
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Hello,

As TeX Live 2009 approaches (too quickly) I'd like to again raise the  
problems of inputenc in XeTeX and LuaTeX. Sorry to add to the  
distractions of the members of the LaTeX3 team who are also busy  
working on the expl3 code...

The problem is clear: LaTeX assumes 8-bit encodings and clobbers the  
use of true UTF8 when newer engines are being used. This is an issue  
because "\usepackage[utf8]{inputenc}" is the first thing that many new  
users do when moving their documents over to the new engine.

Elie Roux and I have been writing packages to attempt to work around  
the problem for LuaTeX and XeTeX, respectively. Our development code  
is here:
   <http://github.com/eroux/lualatex/tree/master>
   <http://github.com/wspr/xetex-inputenc/tree/master>

We'd like inputenc to branch to these packages in the appropriate  
context.

* * *

The LuaTeX version is more robust, but both packages attempt to "do  
the right thing" when they're loaded, keeping UTF8 untouched and  
processing the other encodings as best they can. In the LuaTeX  
package, the original inputenc is still loaded and used for  
transforming the chars.

In the XeTeX package, I take the lazy way out and simply read the  
files in the different encoding, bypassing inputenc entirely.

Neither package attempts to do anything with the macros defined by  
inputenc.sty or utf8ienc.dtx, but the main one we'd be interested in,  
\DeclareUnicodeCharacter, would be easy enough to re-write for  
"unicodeTeX".

However, neither package is really suitable for maintenance by the  
LaTeX3 Project. Elie's macros for supporting LuaTeX are obviously very  
new; furthermore, there're a few intertwined packages in his code tree  
that wouldn't be appropriate, yet, to add to the stable LaTeX2e code.  
And the XeTeX version isn't robust enough because there's no way to  
reliably detect or branch based on \input. (It only works for packages  
that use \@input for auto-generated files.)

* * *

Therefore, my proposal is to remove the responsibility of inputenc  
from the LaTeX2e code for XeTeX and LuaTeX. I'm not sure exactly the  
best way to achieve this; I'm going to recommend hard-coding it into  
inputenc since that's the simplest option for now. E.g., at the top of  
inputenc.sty:

	% check for XeTeX:
	\begingroup\expandafter\expandafter\expandafter\endgroup
	\expandafter\ifx\csname XeTeXversion\endcsname\relax\else
	  \IfFileExists{xetex-inputenc.sty}{%
		  \DeclareOption*{\PassOptionToPackage{\CurrentOption}{xetex- 
inputenc}}
		  \ProcessOptions
		  \RequirePackage{xetex-inputenc}
		  \expandafter\endinput
	  }{}
	\fi

and something equivalent for LuaTeX.

Then the LaTeX team can "ignore" the problem for now (since it's plain  
that LaTeX2e will never natively support unicode), and leave the  
problem in the hands of the developers who are actively writing macro  
packages for the new engines.

Like I said above, I'd like to resolve this for TeXLive 2009, and with  
such a simple patch to inputenc I think we can do it.

Will


--Apple-Mail-6--966340416
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGITCCAtow
ggJDoAMCAQICECN4qE5kBXLk2f/jVDfSZPwwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA5MDEyOTA1NDkxNVoXDTEwMDEyOTA1NDkx
NVowQjEfMB0GA1UEAxMWVGhhd3RlIEZyZWVtYWlsIE1lbWJlcjEfMB0GCSqGSIb3DQEJARYQd3Nw
cjgxQGdtYWlsLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL0BeSiAbKuqxeRN
p2qn/m8ZL+xawr/WXyPgEF0FipWgRe9l3sMXcFHokcUu0xOc97R7xkUsGcQ8EyybGHuWey6x7X1Y
xJZXnoAxqcaG+eREytoYGMIKs6BhEEogLVb2ERw3lQNVnOzanSFeGo8suMAN4zzCtqAjJiA1ph7h
1pksTgECYK5EiIZbFsB6zSDa8crNk404z1CfIA6YO8ezvjbDda+D0r8NU2tq9WS9F5IaG+bW71Ya
JegEcSZ+WF6Z+fs2MUMCLLu8n50Er0nuy4dxOmkdMRNfbeaM39dsEwjAAgcQnvPNmlJ215nZWQRH
49YowtSBOYUYq0ZylWRE6x8CAwEAAaMtMCswGwYDVR0RBBQwEoEQd3NwcjgxQGdtYWlsLmNvbTAM
BgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBABaRP8+PDYpKIRGlFgjs1HvMmJnqu4reSqp+
ulv0zJZIjIbX/sLbIsnecl9nycHfhubPdc+hDfpCqNZ2+NGQHwwoyuDl7KOdTY0BDPp3eJLio7ob
EYEr0H8rFwqfx2LWJ0G6nMhNEjLvs7sFKyriSpk++TWJnnsf86xai5m0tlOwMIIDPzCCAqigAwIB
AgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2Fw
ZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE
CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUu
Y29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMCWkExJTAjBgNV
BAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEpjxVc1X7
TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAKMNcCY1osiRVwjt3J8CuFWqo/
cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7n2XRxSpUhQ9IBH+nttE8YQRA
HmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAwQwYDVR0fBDwwOjA4oDagNIYy
aHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0P
BAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG
9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8
/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQ
Gls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAxAwggMMAgEBMHYwYjELMAkGA1UEBhMCWkExJTAj
BgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJz
b25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhAjeKhOZAVy5Nn/41Q30mT8MAkGBSsOAwIaBQCgggFv
MBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDMxNjA2NTMwMVow
IwYJKoZIhvcNAQkEMRYEFOq8rHcgsT9GMuLMExeT2YVNkMXbMIGFBgkrBgEEAYI3EAQxeDB2MGIx
CzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYD
VQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQI3ioTmQFcuTZ/+NUN9Jk
/DCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29u
c3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNz
dWluZyBDQQIQI3ioTmQFcuTZ/+NUN9Jk/DANBgkqhkiG9w0BAQEFAASCAQCKvgsRvg6PLE9EFGoD
xM0a1z3lRmpxE7ssxxnXZOII03WiaAIC1kNPLanXC9EQ6Uy+AuKaC+JHroYfSGg6dl53rt8yGbXZ
Wy000jkbJLpB4QRkv16s9TlScfSssvcgaMMW0rFi0vnzn1MMHOf1L4LE8yRFWXstKrMqWlcGFoeN
VgnZZrTb0bQ+eA/+4Nl7vtbjeIyvqXot/TWrNhE0d830lGTeKZIeVINEdirNh3UDSr3ag7dcRDgk
+q04fjjZMJ1Gpe+zPilGDdOELSmQHb6ZY3f009oJzJYpX55/8CCe5HtlkVAV/tgi5RTJT4Sy47Xo
ij1gA+620OE3H2ADJuiXAAAAAAAA

--Apple-Mail-6--966340416--