yymao / adstex Goto Github PK
View Code? Open in Web Editor NEWAutomated generation of NASA ADS bibtex entries directly from citation keys in your TeX source files
Home Page: https://github.com/yymao/adstex#adstex
License: MIT License
Automated generation of NASA ADS bibtex entries directly from citation keys in your TeX source files
Home Page: https://github.com/yymao/adstex#adstex
License: MIT License
This is not adstex
problem that could have an adstex
-specific solution. The ADS servers (api.adsabs.harvard.edu) apparently got a new SSL certificate that my operating system (unlike my browser) does not recognize. It would be great to add a command line option to adstex
that would tell it to skip certificate validation while connecting, for example as discussed here.
For now I'm getting the following error message:
$ adstex nher.tex
------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): nher.bib
------------------------------------------------------------
Overwritting existing string for key: june.
Traceback (most recent call last):
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/kirx/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/kirx/.local/bin/adstex", line 33, in <module>
sys.exit(load_entry_point('adstex==0.3.12', 'console_scripts', 'adstex')())
File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 396, in main
bibcode_new = entry2bibcode(bib.entries_dict[key])
File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 261, in entry2bibcode
id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
File "/home/kirx/.local/lib/python3.8/site-packages/adstex.py", line 165, in id2bibcode
return next(s).bibcode
File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 501, in __next__
self.execute()
File "/home/kirx/.local/lib/python3.8/site-packages/ads/search.py", line 533, in execute
self.session.get(self.HTTP_ENDPOINT, params=self.query)
File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.adsabs.harvard.edu', port=443): Max retries exceeded with url: /v1/search/query/?q=identifier%3A%222013MNRAS.433.1991R%22&fl=id&fl=bibcode&sort=score+desc%2Cid+desc&cursorMark=%2A&rows=50 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1147)')))
Hi @yymao,
Really interesting project. I was wondering if you could support an even simpler use case:
If I give the program a ADS-derived bib file, it could look for updates to the entries, for example by adsurl entry. In that case, there is no TEX file.
It would help me update stale entries which I created based on arxiv submissions.
Cheers,
Johannes
It should be able to understand the latex "%" style comments and ignore (unresolvable) cites in those.
Would it be possible to add support for @software
bibtex records for citing codes from the Astrophysics Source Code Library (ASCL)? For example, I'm citing the ASCL record for Fermitools
that I got from the ADS:
@software{2019ascl.soft05011F,
author = {{Fermi Science Support Development Team}},
title = "{Fermitools: Fermi Science Tools}",
howpublished = {Astrophysics Source Code Library, record ascl:1905.011},
year = 2019,
month = may,
eid = {ascl:1905.011},
adsurl = {https://ui.adsabs.harvard.edu/abs/2019ascl.soft05011F},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
Unfortunately, when running on my paper source the adstex
reports
Entry type software not standard. Not considered.
and ignores this record, so I have to insert it manually in the .bib
file every time I re-reun adstex
, which is a minor, but seemingly unnecessary inconvenience.
As noted in the readme, by default adstex checks for updates and it runs kind of slow as a result of doing many ADS queries serially. Parallelizing this step is relatively easy and would speed things up dramatically.
I have implemented parallelization using joblib in my fork (dwijn/adstex) and it seems to work properly, though I have not tested my code extensively.
I just cloned adstex and installed it, and managed to proceed with the ADSbeta registration/token retrieval with chromium, but on firefox I get a 404.html not found error.... Any idea? It might have to do with a local config, but I did not find which one (nor do I see anything obvious on the chromium side).
If no "-o" option is specified in the command line, just find the bibliography command in the source and pull it from there.
adstex (search_keys
?) failed to identify Diemer2018
as a citation in the following example:
\citet[with implementation by \citealt{Diemer2018}]{watson13}
natbib
citations such as
\citep[see also][]{Aauthor2019,Bauthor2020}
are not correctly recognized if a line-break occurs. Some authors/text editors might hard-wrap this citation at the space:
Lorem ipsum dolor sit amet, consectetur adipiscing \citep[see
also][]{Aauthor2019,Bauthor2020} eget risus sollicitudin neque
imperdiet suscipit.
adstex
fails to detect those bib keys.
When I run
adstex main.tex -o main.bib -r mylib.bib --merge-other --force-regenerate
It failed to update the following items from the existing mylib.bib file (which is exported from zotero)
@article{Simon2019,
title = {The {{Faintest Dwarf Galaxies}}},
language = {en},
journal = {arXiv e-prints},
url = {https://ui.adsabs.harvard.edu/abs/2019arXiv190105465S/abstract},
author = {Simon, Joshua D.},
month = jan,
pages = {arXiv:1901.05465},
year = {2019}
}
@article{Richings2018,
title = {Subhalo Destruction in the {{Apostle}} and {{Auriga}} Simulations},
language = {en},
journal = {ArXiv e-prints},
url = {https://ui.adsabs.harvard.edu/\#abs/arXiv:1811.12437},
author = {Richings, Jack and Frenk, Carlos and Jenkins, Adrian and Robertson, Andrew},
month = nov,
pages = {arXiv:1811.12437},
year = {2018}
}
Usually, the arxiv number should be given as eprint
for ads entries. Not sure why these items are not in the standard format. I can fix it by adding the following block at the end of the function entry2bibcode
.
if "pages" in entry and entry['pages'].lower().startswith('arxiv:'):
s = fixedAdsSearchQuery(arxiv=entry["pages"].split(':')[1], fl=["bibcode"])
try:
return next(s).bibcode
except StopIteration:
pass
I'm satisfied with my solution for now. If you feel it could be a general problem for others, you can take it into your code. Thanks.
AGU journals insist on apacite-style citations, which adstex does not appear to recognise. From their latex template:
`
% Reference citation instructions and examples:
% Please use ONLY \cite and \citeA for reference citations.
% \cite for parenthetical references
% ...as shown in recent studies (Simpson et al., 2019)
% \citeA for in-text citations
% ...Simpson et al. (2019) have shown...
%
%
%...as shown by \citeA{jskilby}.
%...as shown by \citeA{lewin76}, \citeA{carson86}, \citeA{bartoldy02}, and \citeA{rinaldi03}.
%...has been shown \cite{jskilbye}.
%...has been shown \cite{lewin76,carson86,bartoldy02,rinaldi03}.
%... \cite <i.e.>[]{lewin76,carson86,bartoldy02,rinaldi03}.
%...has been shown by \cite <e.g.,>[and others]{lewin76}.
%
% apacite uses < > for prenotes and [ ] for postnotes
% DO NOT use other cite commands (e.g., \citet, \citep, \citeyear, \citealp, etc.).
% \nocite is okay to use to add references from your Supporting Information
%`
I am currently using the following workaround to generate a temporary .tex
file to run adstex against:
sed 's/citeA/cite/g' ms_grl.tex | sed 's/\\cite<[^>]*>/\\cite/g' > ms_tmp.tex
i.e. replacing \citeA
with \cite
and removing the <>
-delimited prenotes.
When writing proposals, I sometimes save space with the bibentry package. In that case, the bibliography file is marked with the \nobibliography{filename}
command instead of \bibliography{filename}
and references are inserted with \bibentry{key}
instead of \cite{key}
. I can make the bibentry syntax work by modifying the definitions for _re_bib
and _re_cite
in the code, but I can't figure out how to make a regex that would find both bibentry
and cite
or biblography
and nobiblography
with the same regex. You are clearly a wizzard here, so maybe you can add that.
Stumped by this error
$ adstex test.tex
------------------------------------------------------------
Auto-identifying bibtex files...
Main bibtex source (output file): bib.bib
------------------------------------------------------------
Traceback (most recent call last):
File "/home/msyriac/.local/bin/adstex", line 8, in <module>
sys.exit(main())
File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 394, in main
bibcode_new = entry2bibcode(bib.entries_dict[key])
File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 259, in entry2bibcode
id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
File "/home/msyriac/.local/lib/python3.8/site-packages/adstex.py", line 164, in id2bibcode
return next(s).bibcode
File "/home/msyriac/.local/lib/python3.8/site-packages/ads/search.py", line 501, in __next__
self.execute()
File "/home/msyriac/.local/lib/python3.8/site-packages/ads/search.py", line 532, in execute
self.response = SolrResponse.load_http_response(
File "/home/msyriac/.local/lib/python3.8/site-packages/ads/base.py", line 94, in load_http_response
raise APIResponseError(http_response.text)
ads.exceptions.APIResponseError: '{"responseHeader":{"status":400,"QTime":0,"params":{"q":"identifier:10.1051/0004-6361:20066170","fl":"id,bibcode","cursorMark":"*","start":"0","internal_logging_params":"X-Amzn-Trace-Id=Root=1-5e4abdd3-c8b04fc0540672306bd26010","sort":"score desc,id desc","rows":"50","wt":"json"}},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","java.lang.Exception"],"msg":"org.apache.solr.search.SyntaxError: INVALID_SYNTAX_CANNOT_PARSE: Syntax Error, cannot parse identifier:10.1051/0004-6361:20066170: The parser reported a syntax error, antlrqueryparser hates errors! ","code":400}}\n'
that can be reproduced whenever a bib entry with a very specific DOI exists:
@article{Test,
author = {{Test.}},
doi = {10.1051/0004-6361:20066170},
}
(All other cites in my tex file worked, so thanks for making this!)
Lines 394 to 398 in 5723da4
When key_exists_in_others
is True and key_exists
is False, the procedure will crash at bibcode = extract_bibcode(bib.entries_dict[key])
, because bib.entries_dict
does not have key
there.
Thanks for writing this code! I've been trying it out for the past week and it was working fine, but now I've encountered the following error when it reaches one of the references. Is there a way to determine which reference causes this? Perhaps it would be useful to have a way to skip to the next reference when such an error is encountered rather than quit the program?
Traceback (most recent call last):
File "/astro/mark/anaconda2/envs/ads/bin/adstex", line 10, in
sys.exit(main())
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 395, in main
bibcode_new = entry2bibcode(bib.entries_dict[key])
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 260, in entry2bibcode
id_this = id2bibcode(unquote(entry[field_name]), possible_id_types)
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/adstex.py", line 165, in id2bibcode
return next(s).bibcode
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/search.py", line 501, in next
self.execute()
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/search.py", line 532, in execute
self.response = SolrResponse.load_http_response(
File "/astro/mark/anaconda2/envs/ads/lib/python3.10/site-packages/ads/base.py", line 94, in load_http_response
raise APIResponseError(http_response.text)
ads.exceptions.APIResponseError: '{\n "error": "Not found"\n}\n'
Hmm adstex
appears to run fine, but new entries are not showing up in the .bib
file.
$ adstex --no-update ms.tex -o ms.bib
Overwriting existing string for key: june.
conroy2010: EXISTING
stsci2020: EXISTING
harris2016: EXISTING
bradley2021: EXISTING
foreman-mackey2013: EXISTING
harris2018: EXISTING
astropy:2018: EXISTING
aguerri2020: EXISTING
lim2020: EXISTING
simionescu2011: EXISTING
janssens2019: EXISTING
harris1981: EXISTING
vandokkum2015: EXISTING
schlafly2011: EXISTING
deustua2018: EXISTING
stetson1987: EXISTING
harris2020: EXISTING
bertin1996: EXISTING
conroy2009: EXISTING
sirianni2005: EXISTING
saha2011: EXISTING
astropy:2013: EXISTING
vanderburg2016: EXISTING
------------------------------------------------------------
Resolving keys that do not contain identifiers...
------------------------------------------------------------
------------------------------------------------------------
Choose one entry from below for <erwin2015> (most cited at the end)
------------------------------------------------------------
[5] 2015AAS...22544505E (cited 0 times)
Erwin, Peter; Saglia, Roberto; Thomas, Jens; Opitsch, Michael et al.
SMBH Measurements and Host-Galaxy Correlations: Ellipticals, Bulges, Pseud
[4] 2015DPS....4721017E (cited 0 times)
Erwin, Justin; Koskinen, Tommi T.; Yelle, Roger V.
Radiative equilibrium and escape of Pluto's atmosphere
[3] 2015IAUS..309..359E (cited 2 times)
Erwin, Peter; Saglia, Roberto; Thomas, Jens; Fabricius, Maximilian et al.
Using 3D Spectroscopy to Probe the Orbital Structure of Composite Bulges
[2] 2015MNRAS.446.4039E (cited 104 times)
Erwin, Peter; Saglia, Roberto P.; Fabricius, Maximilian et al.
Composite bulges: the coexistence of classical bulges and discy pseudo-bul
[1] 2015ApJ...799..226E (cited 208 times)
Erwin, Peter
IMFIT: A Fast, Flexible New Program for Astronomical Image Fitting
------------------------------------------------------------
Choose one entry from above for <erwin2015>
------------------------------------------------------------
ENTER choice (if no matches, ENTER 0 to skip or ENTER an identifier): 1
erwin2015: NEW ENTRY => 2015ApJ...799..226E
------------------------------------------------------------
Building new bibtex file, please wait...
------------------------------------------------------------
------------------------------------------------------------
Done!
------------------------------------------------------------
adstex
is definitely writing a new bib file as the timestamp is updated, but it does not contain the new entry.
$ grep -i erwin2015 ms.bib
$
Recently upgraded to macOS Sonoma in case that's a clue. It also means that prior to this I was likely running the version of adstex
that was available in Dec 2022.
There should be a ~/.adstexrc file or something that would allow aliases for commonly used papers. E.g Planck18 should resolve to whatever the arxiv number is (but the search would be performed to ensure published papers become published once published :), i.e pure key-value resolution).
For some reason the call at the end of main
to check for a newer version gets stuck. Going to the website in a browser actually works, so I'm not sure what the issue is. Perhaps it's related to a missing newline as the data is received but the call still hangs.
In any case, what works for me and what would help such a situation (or in the rare case where the site is down) would be to add a timeout to the requests.get
call like this:
try:
latest_version = StrictVersion(requests.get('https://pypi.python.org/pypi/adstex/json', timeout=0.1).json()['info']['version'])
except (requests.RequestException, KeyError, ValueError):
pass
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.