sasozivanovic / memoize Goto Github PK

This is probably a misconfiguration somewhere on my machine (MikTeX 23.4 on Ubuntu 22.04.3 LTS) but I can't get none of the extraction methods to work.

Perl:

2023-10-23 19:30:38,868+0200 INFO  memoize-extract - this process (13200) started by sh in directory <path> with command line: memoize-extract.pl --mkdir ./<filename>.memo.dir
2023-10-23 19:30:38,871+0200 FATAL memoize-extract.core - The '{filename}' configuration file contains no record for '{name}'.
2023-10-23 19:30:38,871+0200 FATAL memoize-extract.core - Data: name="memoize-extract", engine="perl", filename="scripts.ini"
2023-10-23 19:30:38,871+0200 FATAL memoize-extract.core - Source: Libraries/MiKTeX/Core/Session/runperl.cpp:51
2023-10-23 19:30:38,871+0200 FATAL memoize-extract - The 'scripts.ini' configuration file contains no record for 'memoize-extract'.
2023-10-23 19:30:38,871+0200 FATAL memoize-extract - Info: name="memoize-extract", engine="perl", filename="scripts.ini"
2023-10-23 19:30:38,871+0200 FATAL memoize-extract - Source: Libraries/MiKTeX/Core/Session/runperl.cpp
2023-10-23 19:30:38,871+0200 FATAL memoize-extract - Line: 51

Python:

2023-10-23 19:31:08,219+0200 INFO  memoize-extract - this process (13232) started by sh in directory <path> with command line: memoize-extract.py --mkdir ./<filename>.memo.dir
2023-10-23 19:31:08,222+0200 FATAL memoize-extract.core - The '{filename}' configuration file contains no record for '{name}'.
2023-10-23 19:31:08,222+0200 FATAL memoize-extract.core - Data: name="memoize-extract", engine="python", filename="scripts.ini"
2023-10-23 19:31:08,222+0200 FATAL memoize-extract.core - Source: Libraries/MiKTeX/Core/Session/runperl.cpp:51
2023-10-23 19:31:08,222+0200 FATAL memoize-extract - The 'scripts.ini' configuration file contains no record for 'memoize-extract'.
2023-10-23 19:31:08,222+0200 FATAL memoize-extract - Info: name="memoize-extract", engine="python", filename="scripts.ini"
2023-10-23 19:31:08,222+0200 FATAL memoize-extract - Source: Libraries/MiKTeX/Core/Session/runperl.cpp
2023-10-23 19:31:08,222+0200 FATAL memoize-extract - Line: 51

If I'm checking a scripts.ini in ~/.miktex/texmfs/install/miktex/config it contains – under [perl] – the lines

memoize-clean.pl=scripts/memoize/memoize-clean.pl
memoize-extract.pl=scripts/memoize/memoize-extract.pl

(And the same but with .py for [python].)

Note that the text before the = includes the ending .pl even though the error message indicates it seems to look for memoize-extract without the .pl. I can't edit this file without the signature check failing, of course.

Running

perl ~/.miktex/texmfs/install/scripts/memoize/memoize-extract.pl <jobname>.mmz

is sucessful, though.

TeX:

Wille method=tex, extract=sh gives me

pdftex -halt-on-error -interaction=batchmode -jobname "./<filename>.memo.dir/14C9FE4BC0F3D901F310541C7F247C89-E778DCCCB8AAB0BBD3F6CFEEFD2421F8"  "\def \fromdocument {""<filename>.pdf}\def \pagenumber {1}\def \expectedwidth {229.90034pt}\def \expectedheight {186.89198pt}\def \logfile {<filename>.mmz.log}\def \warningtemplate {\noexpand \PackageWarning {memoize}{\warningtext }}\def \mmzpdfmajorversion {1}\def \mmzpdfminorversion {5}\input memoize-extract-one"

running this will return eventually:

2023-10-23 19:33:48,943+0200 FATAL pdftex.core - File name too long: path="./\def \fromdocument {<filename>.pdf}\def \pagenumber {1}\def \expectedwidth {229.90034pt}\def \expectedheight {186.89198pt}\def \logfile {<filename>.mmz.log}\def \warningtemplate {\noexpand \PackageWarning {memoize}{\warningtext }}\def \mmzpdfmajorversion {1}\def \mmzpdfminorversion {5}\input memoize-extract-one.tex"

So, PDFTeX somehow interprets the last argument as a filename instead of TeX source?

Some shaded nodes do not externalize correctly, i.e. they externalize without shading. Here's a MWE demonstrating the problem. After the first compilation, the externalized page (page 1) features a node without shading, even if the same node in the regular output (page 2) is shaded as it should be. On subsequent compilations, memoize uses the externalized picture, so the node appears without shading even in the regular output.

\documentclass{article}
\usepackage{memoize}
\usepackage{tikz}
\begin{document}

\begin{tikzpicture}
  \node[top color=white,bottom color=black!20]{This node should be shaded.};
\end{tikzpicture}

\end{document}

This issue affects a node iff the shading used by the node was not yet used as a part of the output on a regular page. That is, once a particular shading, say top color=white,bottom color=black!20, was used in some node that was already included in some page of the regular output, it will be externalized correctly. After the first compilation of the document below, the first externalized picture comes out wrong (page 1), but the second one (page 3) comes out shaded as expected.

\documentclass{article}
\usepackage{memoize}
\usepackage{tikz}
\begin{document}

\begin{tikzpicture}
  \node[top color=white,bottom color=black!20]{This node should be shaded.};
\end{tikzpicture}

\newpage

\begin{tikzpicture}
  \node[top color=white,bottom color=black!20]{This node turns out ok.};
\end{tikzpicture}

\end{document}

Analysis. When a shading is first used, PGF inserts some specials to define it. The specials are inserted as a part of the \output routine. But as memoize ships out the externalized graphics immediately (so, before the regular output), the shading is used before it is defined.

Interestingly, this leads to the problem only for XeTeX engine. It appears that this sort of "forward reference" presents no problem for pdfTeX (and LuaTeX). In particular, the problem seems to be related to the XDV -> PDF conversion. This can be observed by manually compiling the above (say, first) document in two stages:

xelatex --no-pdf test.tex
xdvipdfmx test.xdv

The second commands produces the following warning:

xdvipdfmx:warning: Error locating image file "pgfshade1"
xdvipdfmx:warning: Specified (image) object doesn't exist: pgfshade1
xdvipdfmx:warning: Interpreting special command uxobj (pdf:) failed.
xdvipdfmx:warning: >> at page="1" position="(87.6767, 28.8323)" (in PDF)
xdvipdfmx:warning: >> xxx "pdf:uxobj @pgfshade1"

If the pages are reordered during XDV -> PDF conversion, the warning disappears and the problematic node comes out as shaded.

xdvipdfmx -s 2,1 testsh.xdv

I can see two approaches towards the resolution of this issue.

1. Modify memoize so that externalized pages are shipped out only after their regular output counterparts are. I can see some downsides of this approach. First, the package would need to keep (an unbounded number of) boxes containing externalized graphics around until the page containing them is finnaly shipped out. Second, the logic implementing the delayed shipout could easily turn out quite complex, especially as externalized graphics might occur in floats. This complexity could easily lead to unwanted interactions with other packages, especially as implementing it would require hacking into the \output routine. A variant of this approach would keep all the externalized stuff around until the end of the document. While this would be easier to implement, it would increase the memory requirement --- I'd say that for some documents, fatally so.

2. Modify xdvipdfmx to properly resolve "forward references". I have no idea how doable this might be, or how it might affect the compilation time.

In the Extern extraction section there are the following paragraphs:

In LaTeX and ConTeXt, the selected method is executed at the end of loading the package.
Afterwards, the key is disabled. If you really need to invoke extraction in the preamble, or again,
write extract/<method>.

When invoked from a \mmzset in the document preamble, this key immediately executes the given
extraction method. In the preamble incarnation, the key may be invoked without a value to
execute the previously selected method.

I am not sure what the takeaway from the last statement should be for the end user. In LaTeX, \mmzset cannot be invoked like that, because it happens inside of \includepackage and the use of this key without an argument is disabled afterwards, as explained in the first paragraph.

It seems this package is not available on CTAN. It looks nice (I haven't tried it yet, only discovered it today from pgf-tikz/pgf#758 (comment)), but it seems like it would be nice to have on CTAN

Beamer frames appear to be a tempting target for memoization, but

• a mysterious error is produced without capture=vbox
• with capture=vbox an empty frame appears before a memoized frame, with the following appearing in the log:

Underfull \vbox (badness 10000) has occurred while \output is active []
Overfull \vbox (13.6pt too high) has occurred while \output is active []

Here's what I tried:

\RequirePackage[record=makefile,extract=no,auto={frame}{memoize,capture=vbox}]{memoize}
\documentclass{beamer}
\begin{document}
\begin{frame}
\end{frame}
\end{document}

What do you think, is it a worthwhile idea? Thank you!

The commands \(Get|Show)Document(Command|Environment)ArgSpec and \ArgumentSpecification will be removed from the kernel in the next release, available only if xparse is explicitly loaded. Details in this commit. The commands \GetDocumentArgSpec and \ArgumentSpecification are used in the definition of \advice@CollectArgumentsRaw so I suppose a change should be made before the next LaTeX release.

To use Memoize together with mylatexformat, I need to pass extract=no in the preamble, and explicitly invoke \mmzset{extract/tex} in the document.

To provide a minimal example, the preamble in fmt.tex would be:

\documentclass{article}

\usepackage[trace=true,extract=no]{memoize}
\usepackage{tikz}

and the main document would be:

%&mylatexformat
\csname endofdump\endcsname

\mmzset{extract/tex}

\begin{document}
\begin{tikzpicture}
        \draw (0,0) -- (1,1);
\end{tikzpicture}
\end{document}

Would there be any downside in running extractors at \begin{document} instead of immediately in the preamble?