yymao / adstex Goto Github PK

View Code? Open in Web Editor NEW

167.0 7.0 14.0 149 KB

Automated generation of NASA ADS bibtex entries directly from citation keys in your TeX source files

Home Page: https://github.com/yymao/adstex#adstex

License: MIT License

Python 100.00%

latex astronomy bibliography bibtex-files nasa-ads python

adstex's People

Contributors

Stargazers

Watchers

Forkers

lewiscowper johannesbuchner ninoc syrte gaybro8777 pkgw pmocz knut0815 miao-cc alanxuhaojie dwijn pierfra-rocci jamesaird

adstex's Issues

Bug: Failed to identify embeded citation keys of natbib

adstex (search_keys?) failed to identify Diemer2018 as a citation in the following example:

\citet[with implementation by \citealt{Diemer2018}]{watson13}

Command line option to skip SSL certificate validation while connecting to api.adsabs.harvard.edu

This is not adstex problem that could have an adstex-specific solution. The ADS servers (api.adsabs.harvard.edu) apparently got a new SSL certificate that my operating system (unlike my browser) does not recognize. It would be great to add a command line option to adstex that would tell it to skip certificate validation while connecting, for example as discussed here.

For now I'm getting the following error message:

$ adstex nher.tex

------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): nher.bib

------------------------------------------------------------
Overwritting existing string for key: june.
Traceback (most recent call last):
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kirx/.local/bin/adstex", line 33, in <module>
    sys.exit(load_entry_point('adstex==0.3.12', 'console_scripts', 'adstex')())
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 396, in main
    bibcode_new = entry2bibcode(bib.entries_dict[key])
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 261, in entry2bibcode
    id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
  File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 165, in id2bibcode
    return next(s).bibcode
  File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 501, in __next__
    self.execute()
  File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 533, in execute
    self.session.get(self.HTTP_ENDPOINT, params=self.query)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))

Syntax error from a very specific DOI

Stumped by this error

$ adstex test.tex 

------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): bib.bib

------------------------------------------------------------
Traceback (most recent call last):
  File "/home/msyriac/.local/bin/adstex", line 8, in <module>
    sys.exit(main())
  File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 394, in main
    bibcode_new = entry2bibcode(bib.entries_dict[key])
  File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 259, in entry2bibcode
    id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
  File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 164, in id2bibcode
    return next(s).bibcode
  File "/home/msyriac/.local/lib/python3.8/site-packages/ads/search.py", line 501, in __next__
    self.execute()
  File "/home/msyriac/.local/lib/python3.8/site-packages/ads/search.py", line 532, in execute
    self.response = SolrResponse.load_http_response(
  File "/home/msyriac/.local/lib/python3.8/site-packages/ads/base.py", line 94, in load_http_response
    raise APIResponseError(http_response.text)
ads.exceptions.APIResponseError: '{"responseHeader":{"status":400,"QTime":0,"params":{"q":"identifier:10.1051/0004-6361:20066170","fl":"id,bibcode","cursorMark":"*","start":"0","internal_logging_params":"X-Amzn-Trace-Id=Root=1-5e4abdd3-c8b04fc0540672306bd26010","sort":"score desc,id desc","rows":"50","wt":"json"}},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","java.lang.Exception"],"msg":"org.apache.solr.search.SyntaxError: INVALID_SYNTAX_CANNOT_PARSE: Syntax Error, cannot parse identifier:10.1051/0004-6361:20066170: The parser reported a syntax error, antlrqueryparser hates errors! ","code":400}}\n'

that can be reproduced whenever a bib entry with a very specific DOI exists:

@article{Test,                                                                                                                        
 author = {{Test.}},                                                                                                                  
 doi = {10.1051/0004-6361:20066170},                                                                                                  
}

(All other cites in my tex file worked, so thanks for making this!)

Bug when key exists in others

adstex/adstex.py

Lines 394 to 398 in 5723da4

    
           if (key_exists and args.update) or ( 
        
               key_exists_in_others and args.merge_other and args.update 
        
           ): 
        
               bibcode = extract_bibcode(bib.entries_dict[key]) 
        
               bibcode_new = entry2bibcode(bib.entries_dict[key])

When key_exists_in_others is True and key_exists is False, the procedure will crash at bibcode = extract_bibcode(bib.entries_dict[key]), because bib.entries_dict does not have key there.

Parallelize updates

As noted in the readme, by default adstex checks for updates and it runs kind of slow as a result of doing many ADS queries serially. Parallelizing this step is relatively easy and would speed things up dramatically.
I have implemented parallelization using joblib in my fork (dwijn/adstex) and it seems to work properly, though I have not tested my code extensively.

Have a way of keeping aliases for common papers #feature-request

There should be a ~/.adstexrc file or something that would allow aliases for commonly used papers. E.g Planck18 should resolve to whatever the arxiv number is (but the search would be performed to ensure published papers become published once published :), i.e pure key-value resolution).

issue opening the ADS beta site on firefox

I just cloned adstex and installed it, and managed to proceed with the ADSbeta registration/token retrieval with chromium, but on firefox I get a 404.html not found error.... Any idea? It might have to do with a local config, but I did not find which one (nor do I see anything obvious on the chromium side).

Line breaks in citations cause failed capturing of bib keys

natbib citations such as

\citep[see also][]{Aauthor2019,Bauthor2020}

are not correctly recognized if a line-break occurs. Some authors/text editors might hard-wrap this citation at the space:

Lorem ipsum dolor sit amet, consectetur adipiscing \citep[see
also][]{Aauthor2019,Bauthor2020} eget risus sollicitudin neque
imperdiet suscipit.

adstex fails to detect those bib keys.

Not found error

Thanks for writing this code! I've been trying it out for the past week and it was working fine, but now I've encountered the following error when it reaches one of the references. Is there a way to determine which reference causes this? Perhaps it would be useful to have a way to skip to the next reference when such an error is encountered rather than quit the program?

Traceback (most recent call last):
File "/astro/mark/anaconda2/envs/ads/bin/adstex", line 10, in
sys.exit(main())
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 395, in main
bibcode_new = entry2bibcode(bib.entries_dict[key])
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 260, in entry2bibcode
id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 165, in id2bibcode
return next(s).bibcode
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/search.py", line 501, in next
self.execute()
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/search.py", line 532, in execute
self.response = SolrResponse.load_http_response(
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/base.py", line 94, in load_http_response
raise APIResponseError(http_response.text)
ads.exceptions.APIResponseError: '{\n "error": "Not found"\n}\n'

Ignore commented out lines

It should be able to understand the latex "%" style comments and ignore (unresolvable) cites in those.

Update existing bibtex file

Hi @yymao,

Really interesting project. I was wondering if you could support an even simpler use case:

If I give the program a ADS-derived bib file, it could look for updates to the entries, for example by adsurl entry. In that case, there is no TEX file.

It would help me update stale entries which I created based on arxiv submissions.

Cheers,
Johannes

Failed to update arxiv items in existing bib files

When I run
adstex main.tex -o main.bib -r mylib.bib --merge-other --force-regenerate

It failed to update the following items from the existing mylib.bib file (which is exported from zotero)

@article{Simon2019,
  title = {The {{Faintest Dwarf Galaxies}}},
  language = {en},
  journal = {arXiv e-prints},
  url = {https://ui.adsabs.harvard.edu/abs/2019arXiv190105465S/abstract},
  author = {Simon, Joshua D.},
  month = jan,
  pages = {arXiv:1901.05465},
  year = {2019}
}
@article{Richings2018,
  title = {Subhalo Destruction in the {{Apostle}} and {{Auriga}} Simulations},
  language = {en},
  journal = {ArXiv e-prints},
  url = {https://ui.adsabs.harvard.edu/\#abs/arXiv:1811.12437},
  author = {Richings, Jack and Frenk, Carlos and Jenkins, Adrian and Robertson, Andrew},
  month = nov,
  pages = {arXiv:1811.12437},
  year = {2018}
}

Usually, the arxiv number should be given as eprint for ads entries. Not sure why these items are not in the standard format. I can fix it by adding the following block at the end of the function entry2bibcode.

    if "pages" in entry and entry['pages'].lower().startswith('arxiv:'):
        s = fixedAdsSearchQuery(arxiv=entry["pages"].split(':')[1], fl=["bibcode"])
        try:
            return next(s).bibcode
        except StopIteration:
            pass

I'm satisfied with my solution for now. If you feel it could be a general problem for others, you can take it into your code. Thanks.

recognize "bibentry" and "nobibliography"

When writing proposals, I sometimes save space with the bibentry package. In that case, the bibliography file is marked with the \nobibliography{filename} command instead of \bibliography{filename} and references are inserted with \bibentry{key} instead of \cite{key}. I can make the bibentry syntax work by modifying the definitions for _re_bib and _re_cite in the code, but I can't figure out how to make a regex that would find both bibentry and cite or biblography and nobiblography with the same regex. You are clearly a wizzard here, so maybe you can add that.

support apacite-style citations

AGU journals insist on apacite-style citations, which adstex does not appear to recognise. From their latex template:
`
% Reference citation instructions and examples:

% Please use ONLY \cite and \citeA for reference citations.

% \cite for parenthetical references
% ...as shown in recent studies (Simpson et al., 2019)

% \citeA for in-text citations

% ...Simpson et al. (2019) have shown...

%...as shown by \citeA{jskilby}.

%...as shown by \citeA{lewin76}, \citeA{carson86}, \citeA{bartoldy02}, and \citeA{rinaldi03}.

%...has been shown \cite{jskilbye}.

%...has been shown \cite{lewin76,carson86,bartoldy02,rinaldi03}.

%... \cite <i.e.>[]{lewin76,carson86,bartoldy02,rinaldi03}.

%...has been shown by \cite <e.g.,>[and others]{lewin76}.

%
% apacite uses < > for prenotes and [ ] for postnotes
% DO NOT use other cite commands (e.g., \citet, \citep, \citeyear, \citealp, etc.).
% \nocite is okay to use to add references from your Supporting Information
%`

I am currently using the following workaround to generate a temporary .tex file to run adstex against:
sed 's/citeA/cite/g' ms_grl.tex | sed 's/\\cite<[^>]*>/\\cite/g' > ms_tmp.tex
i.e. replacing \citeA with \cite and removing the <>-delimited prenotes.

requests get stuck checking for a newer version

For some reason the call at the end of main to check for a newer version gets stuck. Going to the website in a browser actually works, so I'm not sure what the issue is. Perhaps it's related to a missing newline as the data is received but the call still hangs.

In any case, what works for me and what would help such a situation (or in the rare case where the site is down) would be to add a timeout to the requests.get call like this:

try:
    latest_version = StrictVersion(requests.get('https://pypi.python.org/pypi/adstex/json', timeout=0.1).json()['info']['version'])
except (requests.RequestException, KeyError, ValueError):
    pass

Pull the bibtex file from source if no "-o" is specified #feature-reqest

If no "-o" option is specified in the command line, just find the bibliography command in the source and pull it from there.

bib file not being updated

Hmm adstex appears to run fine, but new entries are not showing up in the .bib file.

$ adstex --no-update ms.tex -o ms.bib
Overwriting existing string for key: june.
conroy2010: EXISTING
stsci2020: EXISTING
harris2016: EXISTING
bradley2021: EXISTING
foreman-mackey2013: EXISTING
harris2018: EXISTING
astropy:2018: EXISTING
aguerri2020: EXISTING
lim2020: EXISTING
simionescu2011: EXISTING
janssens2019: EXISTING
harris1981: EXISTING
vandokkum2015: EXISTING
schlafly2011: EXISTING
deustua2018: EXISTING
stetson1987: EXISTING
harris2020: EXISTING
bertin1996: EXISTING
conroy2009: EXISTING
sirianni2005: EXISTING
saha2011: EXISTING
astropy:2013: EXISTING
vanderburg2016: EXISTING

------------------------------------------------------------
Resolving keys that do not contain identifiers...
------------------------------------------------------------

------------------------------------------------------------
Choose one entry from below for <erwin2015> (most cited at the end)
------------------------------------------------------------
[5] 2015AAS...22544505E (cited 0 times)
    Erwin, Peter; Saglia, Roberto; Thomas, Jens; Opitsch, Michael et al.
    SMBH Measurements and Host-Galaxy Correlations: Ellipticals, Bulges, Pseud

[4] 2015DPS....4721017E (cited 0 times)
    Erwin, Justin; Koskinen, Tommi T.; Yelle, Roger V.
    Radiative equilibrium and escape of Pluto's atmosphere

[3] 2015IAUS..309..359E (cited 2 times)
    Erwin, Peter; Saglia, Roberto; Thomas, Jens; Fabricius, Maximilian et al.
    Using 3D Spectroscopy to Probe the Orbital Structure of Composite Bulges

[2] 2015MNRAS.446.4039E (cited 104 times)
    Erwin, Peter; Saglia, Roberto P.; Fabricius, Maximilian et al.
    Composite bulges: the coexistence of classical bulges and discy pseudo-bul

[1] 2015ApJ...799..226E (cited 208 times)
    Erwin, Peter
    IMFIT: A Fast, Flexible New Program for Astronomical Image Fitting
------------------------------------------------------------
Choose one entry from above for <erwin2015>
------------------------------------------------------------
ENTER choice (if no matches, ENTER 0 to skip or ENTER an identifier): 1
erwin2015: NEW ENTRY => 2015ApJ...799..226E

------------------------------------------------------------
Building new bibtex file, please wait...
------------------------------------------------------------

------------------------------------------------------------
Done!
------------------------------------------------------------

adstex is definitely writing a new bib file as the timestamp is updated, but it does not contain the new entry.

$ grep -i erwin2015 ms.bib
$

Recently upgraded to macOS Sonoma in case that's a clue. It also means that prior to this I was likely running the version of adstex that was available in Dec 2022.

	if (key_exists and args.update) or (
	key_exists_in_others and args.merge_other and args.update
	):
	bibcode = extract_bibcode(bib.entries_dict[key])
	bibcode_new = entry2bibcode(bib.entries_dict[key])