IronPort-PHdr: =?us-ascii?q?9a23=3AomWsJBWhAF/tv92HCpMFS0/Uv03V8LGuZFw894?=
               =?us-ascii?q?YnhrRSc6+q45XlOgnl6O5wiEPSBNuGuehJj+PH6eb9X21F4ZuAvH0GepFWWF?=
               =?us-ascii?q?kJhJZehBQuVeiCD0CzN/v2d2oiBs0XVFJj5XC+PE5WFe7xYlTTpju56jtBUg?=
               =?us-ascii?q?7nO19NL//uUpXXk9zx0umz/5PJZAAdhDe7e7J3IRO1oC3es8AXhc1pLaNigg?=
               =?us-ascii?q?DRrC5ufOJbjXhtOUrVnxv44ZKo+4V/9i1Lp/8738tJUKG/Z6thCLIBU2VgPG?=
               =?us-ascii?q?cy68nm8xLESFLVtEERWWgXjBdERjP9wkqiAcXXtS33/qpn1S2TLJewULY1HD?=
               =?us-ascii?q?Sv7qtmTBXlkihBOzNruGfQwtd9iq5WunfD71R23pLUbYeJNfF/YrKVfNUUQn?=
               =?us-ascii?q?BEV9pQUCoJC521bo8GBe4MdehCqIy1q1wLpBq4TQ6iYYGngj9FgGX32KA80O?=
               =?us-ascii?q?kJHgTP3QhmFNUL8TzVoNjzKKYOQLWt1qCbhT7Hbv5QxXL88N2SKkFn8azKB+?=
               =?us-ascii?q?stN5OOmBpKdUuNlFibpI37MinAzOkJunnBqfdmVKevhm8rpgV7ryKggMsh2e?=
               =?us-ascii?q?yrzsoYzE7J8SJhzcM7P9q9HQR7bNi+EZ5dsSKXH4RzS8ImBWpvvWxpr99O8Y?=
               =?us-ascii?q?7+ZyUMxJk9klTfZvqdco2F5BHufOmUKDZ8wnliffjs4nT6uVjlwer6WM6u1V?=
               =?us-ascii?q?9MpScQidjAuEcG0BnL49SGQP9wpx3z4zuE2gHN5+0BGngayPqLeaYo2aV41o?=
               =?us-ascii?q?AetU3SRGnrnkSzjKKXekgg8OWy5KLrb+euqpiZPo5ywgbwV8Zm0syyB/45Nw?=
               =?us-ascii?q?UIXGWz9uG11bml9kr8CLlHlfw5lKDFvYuSf5xK4P7kRVUMj8B6tEf3BizuyN?=
               =?us-ascii?q?kCmHgbMF9JHXDPx5PkPV3DOrGwDPuyhUitjCY+wvnHOrP7BZCeZnPHkbrnYf?=
               =?us-ascii?q?N88xsGkFt1nY0ZvcsJTOhecaGWOAe5rtHTAx4nPhbh2evjAcQnkJ4YWCePD6?=
               =?us-ascii?q?6VPafYvEWHoO4icIzuLMcYvijwL/896rvgl3g8zBUdcK+z3J4abnO1NvtjKk?=
               =?us-ascii?q?SdJ3Hrh51SdAVC9hp7V+HshFCYBHRTanCoXqcx4DI2II2jBIbHAIuqhfbSuU?=
               =?us-ascii?q?XzVo0TbWdABFeWFH7ufIjRQPYAZhWZJcp5myAFX7ysGMcxkAujvwjgx/97P/?=
               =?us-ascii?q?LZr2cG4In72oE/tIix3VkisCZ5BMOH3ySRQnFoyykWEiQu0vk3oFQhmAzel/?=
               =?us-ascii?q?Ep27oCSYMVv7QTDk87LcKOkbA8UYupHFufJpHRDw/3J7fuSTApEoBvnZlXOx?=
               =?us-ascii?q?w7Q5P6yUqblyuyX+1LyfrRWM1yovqahz+rf4580yqUi/Nn1gVgG5oVczXh3/?=
               =?us-ascii?q?4alUCbBpaVwRrF0f/wKOJEmnaKrTnLzHLS7hgAAEgpDvqDDTZHIRGP5dXhuh?=
               =?us-ascii?q?GbEOToUOV7dFMHkYnbdM4oIpXolQkUHq+yfoSDJTvryyHoQk/XjrKUMNiwIj?=
               =?us-ascii?q?tbgXWbURle1VhUpCfONBBiVH34/SSEV20oTgipOwS1oIwc4DutR0swhVjQPX?=
               =?us-ascii?q?ds3Le05BMZwMekZahIjuAivyEs43VuEVK0zoiQFtyB4gpsfalRZdkw+lgB2W?=
               =?us-ascii?q?+L/wp6doetKaxvnDt8O0x+ol/u2hNrC45BjdlirXUkyxB3IL6Z11UJfi2R3J?=
               =?us-ascii?q?T5MLnaYmfo+xXnZ6nT01DYmNGYn8VHoOw/sEnmtRq1G1AK9nxm15xN3CLZ6M?=
               =?us-ascii?q?yVXUwdVpX+Vktx/B9/5vnbbiQ7+4LIxChsPK2z4Vqgk5oiAOoozArlfs8Kav?=
               =?us-ascii?q?vVUl+vVZdDWo73d71500KkZR8FIu1IobQ5O8q3Lb2b3q/uPOthkD+ih2ld78?=
               =?us-ascii?q?Zx3xHplWI0R+jW0pIC2/zd0BGAUmK2jFqnqMHylIZHaBkdG2O+zW7vA4sbNc?=
               =?us-ascii?q?gQNc4bTHyjJcG63IA0n5n2R3tR70KuHXsD0c6tPAKdNhnzhFIBk0sQpnOjlG?=
               =?us-ascii?q?2zyDk+wFRL5uKPmSfJxerlbh8OPGVGEXJjgVnbKo+xl9kGXUKsYlF1xivg3l?=
               =?us-ascii?q?7zwu1gnIo6N3PaGB4acC3/LmUkWay15OLbMpx/ragwuCASa9ySJFWTS7rzuR?=
               =?us-ascii?q?wfinqxEG5XxTR9fDav6Mygwk5KzVmFJXM2l0L3PMF9wRCFuo7ZTP9VmyIJHW?=
               =?us-ascii?q?x20GaGQFe7ONat8JOfkJKR6u0=3D?=
Mail-Followup-To: LATEX-L@listserv.uni-heidelberg.de
References: <3939613963363954.WA.kellysmith12.21gmail.com@listserv.uni-heidelberg.de>
            <924058d3-d3a3-7e40-d799-fe0bc0e20ffb@morningstar2.co.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Message-ID:  <20200316202737.k6wghfvw3hgtidv6@vento15post8>
Date:         Tue, 17 Mar 2020 09:27:37 +1300
Reply-To:     Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender:       Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From:         Henri Menke <henrimenke@GMAIL.COM>
Subject: Re: Using Lua to Preprocess Unicode Data
To:           LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <924058d3-d3a3-7e40-d799-fe0bc0e20ffb@morningstar2.co.uk>
Precedence: list
Envelope-To: <rainer.schoepf@gmx.net>
Status: R

On 16/03/20, 17:32, Joseph Wright wrote:
> On 16/03/2020 17:01, Kelly Smith wrote:
> > Hello!
> > 
> > I’ve been thinking: since Lua is already involved in the build process,
> > by way of l3build, wouldn’t it be reasonable to use a lua script
> > to preprocess Unicode data into forms that are easily consumed by LaTeX
> > during the format-building process?
> > 
> > Warmly,
> > Kelly
> > 
> 
> It depends on the outcome you are after.
> 
> The original loading method for Unicode data in XeTeX was via a Perl script.
> That created a .tex file containing (for example) catcode data. To update
> the Unicode data, one had to run the Perl script, then send the processed
> files to CTAN. There were two issues. First, that meant that any change
> required active work to not only get the data from Unicode but also to
> manipulate it. Second, and more significant, it was *slower* than just
> reading the files in TeX. (This only became apparent when I wrote some test
> parsers.)
> 
> Now, there is more data being loaded today than when I did that work, and
> some of it is in LuaTeX so could be done Lua-only. It's also possible that
> the Perl script was sub-optimal, or that as part of a general 'install'
> function the time would not really show. However, XeTeX needs the data, so
> one is still looking at having to explicitly pre-process in Lua. Moreover,
> most of the time taken for format-building is not about reading Unicode
> data. With LuaTeX, pre-loading expl3 does cut out a slight 'stall' when
> loading everything for case-changing, but having a LuaTeX and a XeTeX path
> separately is not attractive.

Is there any distribution that doesn't have LuaTeX in the default
installation?  (Apart from exotic things like TeX Live infra-only)  Then
it would be conceivable to just make LuaTeX a hard requirement and
process the Unicode data on the fly instead of going via CTAN.

> The current set-up means that updating the Unicode files is just a question
> of copy-pasting the raw .txt files into a form that CTAN can accept.
> Pre-digesting still leaves us needing some way to co-ordinate between
> packages (format, luaotfload, expl3, specialist stuff), plus with having to
> do the explicit extraction.
> 
> As format-building is all about saving time for 'normal' runs, I'm not
> seeing there is a massive need to speed up the process. I know there is one
> engine in development that doesn't use format files, so that might be a
> place to consider things, but I think we'd need a strong case to alter the
> approach for XeTeX/LuaTeX (pdfTeX, ...).

Are you referring to JSBox?  I doubt that this will every be public.

Cheers, Henri

> Joseph