bib file not found even though known to kpse about biber HOT 68 CLOSED

Hmm, that's the second report of this I've had but currently I can't reproduce it. You are right that it seems that it can't find kpsewhich in the PATH for some reason. Is this on x86 or ARM?

from biber.

It is x86.

from biber.

Just to make sure we're comparing the same thing, can you download onto your desktop the universal binary from here:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/MacOS/biber-darwin_universal.tar.gz/download

(you may well have to clear the security complaint as this binary isn't signed like the TL one - try to open it once, close the complaint window, go into System Preferences->Security & Privacy and you'll see biber mentioned with a "Allow Anyway" button. Click that and on the next attempt to run it, just click "Open" in the popup window and after that it will run normally)

and then run:

~/Desktop/biber --debug --tool biblatex-examples.bib

and then look in the biblatex-examples.bib.blg

to see what's happening?

I just did this on an x86 laptop and it's fine for me.

Also check to make sure kpsewhich is in the PATH of the shell running biber?

from biber.

Many thanks, that downloaded version works. The version on my system with the corresponding invocation fails. kpsewhich runs fine from the same shell.
tlmgr info biber gives

package:     biber
category:    Package
shortdesc:   A BibTeX replacement for users of BibLaTeX
longdesc:    Biber is a BibTeX replacement for users of BibLaTeX. Biber supports full UTF-8, can (re)-encode input and output, supports highly configurable sorting, dynamic bibliography sets and many other features. The CTAN distribution offers a compressed tar archive of the sources, etc., together with "binary" distributions for a variety of platforms. Note: on SourceForge biber is formally named "biblatex-biber", to distinguish it from an earlier (now apparently moribund) project called "biber".
installed:   Yes
revision:    63708
sizes:       src: 1193k, doc: 269k, bin: 98113k
relocatable: No

and tlmgr restore provides

tlmgr restore --dry-run biber
tlmgr restore: dry run, no changes will be made
Available backups for biber: 61867 (2022-06-26 11:52)

So the texlive version has been bumped recently. Should I try to restore this earlier version?

from biber.

Would be interesting to see what difference that makes - I have no idea what is going on here currently - I just released the MacOS stuff as usual and it was pulled into TL.

from biber.

OK, I have now tried that. Unfortunately, restoring the texlive version 61867 doesn't help. Should I/we be asking the texlive maintainers instead? Incidentally reupdating texlive now gives revision number 63752, but it still has the same problem.

from biber.

I think the move to 6308 was just a documentation change as I had to request that yesterday. I just tried the same thing using the TL binary from macTeX TL 2021 and 2022 on both ARM and x86 and I still can't get it to fail. Tried version 61867 and 63708 and both work for me on ARM and x86_64. Very odd.

from biber.

Is there live dependence on perl modules or is everything included in the binary? I notice some parts of my perl installation have updated recently, but I haven't used perl for a long time, so am not sure what to check. The can_run command is from IPC:Cmd. But that works fine

#!/usr/bin/env perl

use IPC::Cmd qw( can_run );
print(can_run('kpsewhich'));

gives

/Library/TeX/texbin/kpsewhich%  

as output

from biber.

There are no live dependencies - it's all internal to the cached perl tree "inside" biber. I was thinking it might be to do with what they do to the binaries to sign them in TL as that one you downloaded was the exact binary they pulled into CTAN and TL. They then have to sign them for MacOS. However, that still doesn't explain why I can run all of this without problems with the TL binaries ...

from biber.

Can you let me know which version of MacOS this is on?

from biber.

I'm on macOS 12.4, Monterey. I'm also using a fully updated TeX Live 2022 originally installed via MacTeX.

By the way, in an earlier email you noted that there were 2 versions of kpsewhich on your system. One was in /opt via MacPorts. The other seemed to be in TeX Live 2021 rather than 2022! I have only one active version of kpsewhich on my PATH.

from biber.

That situation with macports kpsewhich was a red herring I think. It's currently frustrating that I can't reproduce this. I also have TL 2022 via MacTeX on an x86 laptop but it's not 12.4 as it's an older laptop that can't take 12. I have 12.4 on ARM but that doesn't seem to be an issue. Let's see what @AndrewSwann has.

from biber.

I am also on macOS 12.4 Monterey. biber was working fine a week or so ago and I have been running Monterey for some time. I have only one version of kpsewhich on my system: /Library/TeX/texbin/kpsewhich

from biber.

Can you both try deleting the biber cache and trying this again? To do this, run

biber --cache

and just delete the entire directory this returns. Then, be sure to run biber once until it returns so it can unpack the cache again.

from biber.

from biber.

I removed that entire parent directory to the cache and ran biber again, but unfortunately that doesn't solve the problem.

from biber.

Can you try the following and see what happens?

cp /usr/local/texlive/2022/bin/universal-darwin/biber ~/Desktop/biber
codesign --remove-signature ~/Desktop/biber
~/Desktop/biber --tool biblatex-examples.bib

from biber.

Can you also get this small binary:

https://www.dropbox.com/s/kjocvk15eb0sddi/test?dl=0

and run it on your machines with the problem? It just runs the perl code below to see what environment PAR is seeing:

use IPC::Cmd (can_run);
use v5.26;

say 'PATH=' . $ENV{'PATH'};
say can_run "kpsewhich";

from biber.

Moving biber to Desktop as you described, and running worked fine. I have sent you the result of the test script by email.

from biber.

Hmm, did the copy to the Desktop work or was it the codesign signature removal that did the trick?

from biber.

Good point. Just copying to the Desktop without removing the signature works.

from biber.

That is really odd - let's see what happens for @herbs.

from biber.

Just out of interest - what happens if you overwrite the TL binary with the one copied to the Desktop?

from biber.

from biber.

from biber.

and zsh kills it every time now? Sometimes there is a second popup to allow unsigned after the System Preferences auth.

from biber.

One more test then - what about copying the test binary temporarily to /usr/local/texlive/2022/bin/universal-darwin/? I have a feeling that something horrible is happening with MacOS SIP here.

from biber.

Copying it from Desktop back to the texlive location gives the old error

from biber.

No, copying to universal-darwin directory does not help.

from biber.

What output does the "test" binary give when it's in /usr/local/texlive/2022/bin/universal-darwin/? Same as when it is in Desktop?

from biber.

from biber.

I'll get Mr Koch in here if I can - we have already discussed the whole signing/notarizing thing before and I suspect it has to do with this or the SIP enhancements in MacOS 12 on Intel. It is odd however that copying the binary out of the TL tree only works for one of you however. I assume you both have SIP enabled - it would be unless you purposely disabled it - you can check with:

csrutil status

from biber.

Yes, SIP is enabled for me.

from biber.

So, there is a workaround for now by just having the binary outside of TL but before TL in your PATH. This looks like a SIP issue. Wish I could reproduce but it looks like something that >10.15.7 introduced. Have invited Richard Koch here for his opinion.

from biber.

from biber.

That's actually good as this is consistent for both of you now. This must be a SIP thing as it makes no sense in normal UNIX land.

from biber.

Hi,

I have the same issue after updating my texlive a week ago and found this thread while investigating a solution. I'm also on macOS Monterey 12.4, x86, SIP enabled. I have downloaded the binary from Sourceforge and put it in my personal bin folder (which is before texlive in my path). However:

• when I run that biber via the PATH as biber --tool biblatex-examples.bib, it cannot find biblatex-examples.bib
• when I run that biber via an explicit path as ~/bin/biber --tool biblatex-examples.bib, it can find biblatex-examples.bib
• when I run the TexLive installed biber via an explicit path as /Library/TeX/texbin/biber --tool biblatex-examples.bib, it can also find biblatex-examples.bib, and all works as expected

This makes me think it's not a code signing problem of biber but maybe something with how it finds kpsewhich internally? Your test binary was no longer available so I could not test that, but running the code as a perl script gives the expected output.

Also: as Andrew pointed out in the first message, this problem occurs only when the bibtex file is not in the current directory but needs to be found via kpsewhich. If the file is symlinked in the local directory, biber works as expected (this is an easy workaround).

➤ cd ~/tmp
➤ which biber
/Library/TeX/texbin/biber

➤ biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
ERROR - Cannot find 'biblatex-examples.bib'!
INFO - ERRORS: 1

➤ cd ~/bin
➤ tar xzf ~/Downloads/biber-darwin_universal.tar.gz
➤ cd ~/tmp
➤ which biber
/Users/leon/bin/biber

➤ biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
ERROR - Cannot find 'biblatex-examples.bib'!
INFO - ERRORS: 1

➤ ~/bin/biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
INFO - LaTeX decoding ...
INFO - Found BibTeX data source '/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib'
INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable'
INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized'
INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US'
INFO - No sort tailoring available for locale 'en_US'
INFO - Writing 'biblatex-examples_bibertool.bib' with encoding 'UTF-8'
INFO - Output to biblatex-examples_bibertool.bib

➤ /Library/TeX/texbin/biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
INFO - LaTeX decoding ...
INFO - Found BibTeX data source '/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib'
INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized'
INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable'
INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US'
INFO - No sort tailoring available for locale 'en_US'
INFO - Writing 'biblatex-examples_bibertool.bib' with encoding 'UTF-8'
INFO - Output to biblatex-examples_bibertool.bib

➤ which kpsewhich
/Library/TeX/texbin/kpsewhich

➤ kpsewhich biblatex-examples.bib
/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib

cheers,
Leon

from biber.

Hmm @AndrewSwann, @herbs - can you see if using an absolute path works for you? The problem remains that I can't reproduce this on 10.15.7 Intel.

from biber.

Test binary is back:

https://www.dropbox.com/s/nn2ypny2sh0ivf5/test?dl=0

Can you try PATH and absolute calls of this when it's in /Library/TeX/texbin/

from biber.

that test binary only has the arm code, I would need the x86:

➤ file ~/Downloads/test
/Users/leon/Downloads/test: Mach-O 64-bit executable arm64

from biber.

Apologies:
https://www.dropbox.com/s/zu4ka1hpsjdp8wy/test?dl=0

from biber.

from biber.

gotcha!

➤ biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin

➤ /Library/TeX/texbin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

NB: renamed the executable to avoid conflicts with shell built-ins

from biber.

@herbs - yes we preempted that issue and released a new biblatex version last night just after the cut-off for distribution so you will get 3.18a with the fix today.

For an emergency fix, see:

plk/biblatex#1234

from biber.

@leonmoonen - Aha, that's the issue then. How strange - and that doesn't happen with the raw perl if that's in /Library/TeX/texbin?

from biber.

nope, the raw perl script in /Library/TeX/texbin gives the complete path in both cases

from biber.

Can you see what happens to the binary and the script when called through the real (no symlinks) absolute path, e.g:

/usr/local/texlive/2022/bin/x86_64-darwinlegacy/biber-test

from biber.

Through the absolute path, things still break when the executable is found via $PATH, direct calls work as expected

➤ /usr/local/texlive/2022/bin/universal-darwin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ /usr/local/texlive/2022/bin/universal-darwin/raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ fish_add_path -p /usr/local/texlive/2022/bin/universal-darwin

➤ which biber-test
/usr/local/texlive/2022/bin/universal-darwin/biber-test

➤ biber-test
PATH=/usr/local/texlive/2022/bin/universal-darwin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

➤ which raw-biber-test
/usr/local/texlive/2022/bin/universal-darwin/raw-biber-test

➤ raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

from biber.

I can reproduce this on MacOS 10.12 Intel too. Have opened ticket to PAR::Packer as it seems to be something to do with the binary packaging - see mentioned ticket above.

from biber.

Looks like this might be an issue with an older version of PAR::Packer as I can fix the issue by upgrading this on MacOS 10.12. I will be upgrading the MacOS build servers and pushing new versions soon.

from biber.

from biber.

Late to the party here, but I'm having the same problem (regularly installed MacTeX) MacOS 12.4 (Monterey). biber 2.18.

from biber.

from biber.

On Jul 4, 2022, at 1:13 PM, Alan Munn @.***> wrote: Late to the party here, but I'm having the same problem (regularly installed MacTeX) MacOS 12.4 (Monterey). biber 2.18.
Howdy, Just to be clear. Can we assume you are on an Intel system? It seems that the arm64 code runs. I've got one of each, so

Yes, Intel system. Using the full path works, but not the filename alone.

from biber.

It's a problem with the module used to package the binary I believe. I am in the process of getting a new release out with a fix but my main build server is having a few issues I have to deal with first, hopefully this evening.

from biber.

After coincidental and irritating server issues, I have rebuilt the binaries for 2.18 and before I push then to CTAN/TL, they are on SF:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/MacOS/

Can you get the universal binary from here and try it with non-absolute paths?

from biber.

from biber.

@herbs - Did you do the usual unblocking routine via System Preferences? That isn't notarized and so will need to be manually authorized until it's in TL.

from biber.

from biber.

Thanks, this binary works fine for me (Intel Mac).

from biber.

CTAN publishing will be done today and you should see 2.18-1 for MacOS in TL.

from biber.

from biber.

from biber.

Good, at least we know what the problem was and that it's fixed.

from biber.

from biber.

from biber.

from biber.

yeah, it looks like I somehow ended with a mix of old and new versions of biber, and the path changed to prefer an old one...

from biber.