IronPort-PHdr: =?us-ascii?q?9a23=3A7yl2mhd8KSS+ox0IA1cXr9vVlGMj4e+mFxMJ6p?=
               =?us-ascii?q?chl7NFe7ii+JKnBkHE+PFxlwaZDdfB5vZNmrCQrqbhHGwN4JqMtHoPapMKWx?=
               =?us-ascii?q?JWwd4OkVkGB8iIQVb+MOasdzYzScZFT1J9/zekK0laGNrWYlrIpHy04iUOEw?=
               =?us-ascii?q?+lcwFyYPn2ScbJl8ri8ee09tXIZhlQwj+0YLd8NhKz+AfWrcQOgc19MKc+yg?=
               =?us-ascii?q?fhp3xSfeVQynh0LEjJ2R37oNqzr9Z46yoFnfUn+oZbVLniOaQ1SbsNFDM9L2?=
               =?us-ascii?q?U8/9HmrzHGRAqLo2QGCyAYzkcODA/C4xX3GJz2t3Kg5NBw0ySbI8D6CIsMd2?=
               =?us-ascii?q?/+tfVNTxnlwGcdOjow4TuRl8pxyaJWoR6soxZy2YGSbIzHfPx5NrjQe98XXw?=
               =?us-ascii?q?8jFo5YSjBBD4WgboAOE/tJPOBWqJP4rkcPqh32DBelBefmwDtFznHs2qhy3+?=
               =?us-ascii?q?MkGADAlAsuerBG+HbTt9jtNOENS+G6zLXgyDLZafJQ1izh4ZKSNBsm5+yPHP?=
               =?us-ascii?q?pxfcfX1UgzBlbdlFzDzO6tdziR1+kLry2a9786DbPp2zNh8Vso5GTylY82h4?=
               =?us-ascii?q?LEh5wY0AXe+CFw0dxzONu5DU52YNKpFJFdrS7cO4YlJ6FqC2xupis+zaUL/J?=
               =?us-ascii?q?Chey1fgpYg3BPHa7qYaYmH4g7LUeGLJzZ+i2l5cajlwR21t1WjgL6ZNIH8wB?=
               =?us-ascii?q?NRoyxJn8OZ/HIKzBHL6o6YUPpy/luJ3DGV0QHV5fpYKFpu06HcbYMiiO1V9N?=
               =?us-ascii?q?JboQHIGSn4n1/zhamdexA/++Sm3O/gZ63vup6WM4It0FPEP68jm9KyDaEDCi?=
               =?us-ascii?q?ZVAzfJ3+O62fWj50D9SaQQyOAwk+zZvZHWKMARq7S2RQNYgM4l7F6kAjGq3c?=
               =?us-ascii?q?59/zFPJU9ZeB+Bk4niOk3faPH+A/Clhl2wkTBtj/nYN7zlC5/JIzDNirDkNb?=
               =?us-ascii?q?p67kddzkI0w7U9r9pWDaoMOPK1RVXwu9zEJho+KQq1xen8FNxnjMUVXiSSAe?=
               =?us-ascii?q?7RMa/ft0OJ+vN6JuCNYIEPvzOuTpptr/XqjHI/hRocZfzzhcpROC3+RK4/ZR?=
               =?us-ascii?q?nEMjL2j9wMEHkHpF8kQejjmQfETDhSIna/Xq4x4js2Eo3gAYqQI+LlyLGHwi?=
               =?us-ascii?q?q/GYVbI25cDVXZW3XvbIicW7ETciOdItNJlzseU7mnRJQ93Av08gT9jadkZL?=
               =?us-ascii?q?mxmGVQpdf42d556veG3xM17jFsD4KCyWCCT31cm2QSQT4w07tjqFYnjFyEl7?=
               =?us-ascii?q?V7ya89d5Qb97ZCVQE0MoTZxupxBoXpWw7PSdyOTU6vXtSsBTxoFIALztQDYl?=
               =?us-ascii?q?hwF5CZtj6Yg3X4JbgTmvTLHpEw+7OGmWX3JoN9xnfK2a0rgkMpBMdCZyWgge?=
               =?us-ascii?q?Zk+g7fCpSs8Q3Rnru2dakawC/G9XuShWuIskZCVQdsUKLDFXkBb0rSpN796w?=
               =?us-ascii?q?vMVbirQbggNwJAz4aFJM4oIpXxik5aQf74JNnES2e4mmP1GgzRg73QMczlfG?=
               =?us-ascii?q?IS2CibA08B0kgS8XuAKQkiF3KhrmbZX1kMXRrkZ0Lh9/U7qWvuFBdliVHXKR?=
               =?us-ascii?q?c7jvzpp01dn/GXRvIN06hRoioloiQuWku73pfWBtuMqg5rcb9TJ98wszIlnS?=
               =?us-ascii?q?rUsRJwOpu4IuVsnFkbJk51uFjjzxUxFZ9Bnsg2hHYt1gxzL62DzFpbLXWT2t?=
               =?us-ascii?q?btOfeETwu6tADqcKPQ1lzEhZyO/bwT7f0jt1j5lASgF05n7G8+ltcJjj2T4Z?=
               =?us-ascii?q?LFCAdUWpX0GBVSlVAyt/TRZS8z4JnR3HtnPPyvszPM7NkuAfMs1heqe9oGbP?=
               =?us-ascii?q?G0UTTqGshfPPCAbfQwkgLxPBcDJuZT8qEvI8q8Lb2N3+i2P7Q4xWP0vSF8+I?=
               =?us-ascii?q?l4l3m02W95Q+/M0YwCxqvAjA6ATTH9ili6rsntw8ZPYncPHTjnkHW2NMtqfq?=
               =?us-ascii?q?R3OL0zJyKuLsmwnIgsgpfsXztH7gfmCQ9bnsCufhWWYhr22ggCjUk=3D?=
References: <8422310776248060.WA.kellysmith12.21gmail.com@listserv.uni-heidelberg.de>
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
            Thunderbird/68.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-GB
Content-Transfer-Encoding: 8bit
Message-ID:  <d7e60ce4-3285-1410-9429-6d1c983491bf@morningstar2.co.uk>
Date:         Tue, 17 Mar 2020 08:30:54 +0000
Reply-To:     Mailing list for the LaTeX3 project
              <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
Sender:       Mailing list for the LaTeX3 project <LATEX-L@LISTSERV.UNI-HEIDELBERG.DE>
From:         Joseph Wright <joseph.wright@MORNINGSTAR2.CO.UK>
Subject: Re: Using Lua to Preprocess Unicode Data
To:           LATEX-L@LISTSERV.UNI-HEIDELBERG.DE
In-Reply-To:  <8422310776248060.WA.kellysmith12.21gmail.com@listserv.uni-heidelberg.de>
Precedence: list
Envelope-To: <rainer.schoepf@gmx.net>
Status: R

On 16/03/2020 22:54, Kelly Smith wrote:
>> As format-building is all about saving time for 'normal' runs, I'm not
>> seeing there is a massive need to speed up the process. I know there is
>> one engine in development that doesn't use format files, so that might
>> be a place to consider things, but I think we'd need a strong case to
>> alter the approach for XeTeX/LuaTeX (pdfTeX, ...).
>>
>> Joseph
> 
> Sorry, I should’ve clarified: the point of preprocessing the data wouldn’t
> be to speed up anything, instead, the point would be to do complex
> processing that would be very difficult or even impossible in LaTeX.
> 
> For example, if the l3regex module were extended so that precompiled
> regexes could be used as parts of other regexes, then Unicode properties
> could be simply implemented by referring to precompiled regexes whose
> content was created by running filters over the Unicode character database.
> 
> Another example would be processing the very complex XML files that are
> used in supplementary Unicode files, like the Common Locale Data Repository,
> which could help with localization and language-specific date and number
> parsing/formatting.
> 
> This idea of preprocessing could be applied to any complex data set that
> LaTeX3 may need to work with, but I used the example of Unicode data
> because that’s the one that immediately came to mind.

When there is a need to do complex pre-processing, Lua is the obvious 
way to go nowadays, and that's reflected in a number of scripts. The 
issue tends to be not what one does to script extraction, but rather the 
target at the TeX end.

For example, on the regex idea, I suspect performance would be the major 
concern: regex processing is already a lot of work, and I have a feeling 
Bruno would want to optimise how the data were stored inside TeX. (I'm 
not sure where the balance between data extraction and storage lies 
here.) Similarly, a lot of work is done by Javier for babel using the 
CLDR, but many of the outcomes are not amendable to scripting: it's 
about how you set up the TeX 'just right'.

So whilst there is no reason not to use Lua when it works, at the moment 
there are not pressing areas where the pre-processing is the barrier.

Joseph