Automatic citation extraction from URLs - phiresky's blog

[tg-z]

Dec 13, 2019

pandoc-url2cite[1] allows you to instantly and transparently cite most papers directly given only a single URL.

You simply add a URL of a publication, and it will replace that with a real citation in whatever CSL[2] style you want. This means you can avoid dealing with Mendeley[3] or Zotero[4] and keeping your Reference Manager database and bibtex file in sync, especially when collaborating with others.

Here is a minimal example:

minimal.md

Compiling this file with this command

pandoc \
    --filter=pandoc-url2cite --filter=pandoc-citeproc \
    --csl ieee-with-url.csl \
    minimal.md -o minimal.pdf

This results in the following output:
minimal.pdf

For a longer example, you can look at the source of this file itself, which is both a blog post[5], GitHub Readme and LaTeX "paper":

README.pdf

Source README.md - Result README.pdf

Install this package globally using npm install -g pandoc-url2cite.

Then, add --filter=pandoc-url2cite to your pandoc command (before pandoc-citeproc, see the minimal example above).

Alternatively, clone this repo[1] somewhere, then install the dependencies using npm ci install.

If you’re not familiar with writing papers in pandoc, you can refer to e.g. this article[6]. It’s pretty flexible, you can use templates from whatever conference you want, and you can still use inline latex code if you need it (and you are ok with not being able to convert your document to nice HTML or EPUB anymore).

Citation Syntax

url2cite allows multiple ways to cite:

1. (PREFERRED) Use the pandoc citation syntax for citations:

   The authors of [@alexnet] first introduced CNNs to the ImageNet challenge.

   More information about referencing specific pages etc. is in the pandoc manual[7].

   Then add the URLs with the usual "link reference" syntax to the bottom of your document in its own paragraph:

   [@alexnet]: https://...

2. Convert all links to citations

   Add url2cite: all-links to your yaml front matter[8]. This will cause all links in the document to be converted to references.

   You can still blacklist some links by adding no-url2cite to either the CSS class of the link (pandoc-only):

   [foo]([http://example.com](http://example.com/ "Linkify Plus Plus")){.no-url2cite}

   or to the link title:

   [foo]([http://example.com](http://example.com/ "Linkify Plus Plus") "no-url2cite").

The main idea is that usually every piece of research you might want to cite is fully identifiable by an URL - no need to manually enter metadata like author, release date, journal, etc. Citation managers like Zotero already use this and enable you to automatically fetch metadata from a website. But then you still have a citation database somewhere that you may or may not be able to synchronize with different computers, but probably won’t be able to add to the version control of your paper. There’s hacks such as better-bibtex[9] to automatically generate and update diffable bibtex files – But that means you now have two sources of truth, and since the export is one-way this leads to multiple contributors overriding each other’s changes. pandoc-url2cite goes a step further: URLs are directly used as the cite keys, and the "bibliography file" is just an auto-generated intermediary artifact of those URLs.

pandoc-url2cite is based on the work of the Zotero[4] developers. Zotero has a set of "Translators"[10] that are able to extract citation info from a number of specific and general web pages. These translators are written in Javascript and run within the context of the given web site. They are made to be used from the Zotero Connector browser extension, but thankfully there is a standalone Translation Server[11] as well. To avoid the effort required to automatically start and manage this server locally, pandoc-url2cite instead uses a publicly accessible instance of this server provided by Wikipedia with a public REST API[12].

All citation data is cached (permanently) as bibtex as well as CSL to citation-cache.json. This is both to improve performance and to make sure references stay the same forever after the initial fetch, as well as to avoid problems if the API might be down in the future. This also means that errors in the citation data can be fixed manually, although if you find you need to do a lot of manual tweaking you might again be better off with Zotero.

1. Currently, extracting the metadata from direct URLs of full text PDFs does not work, so you will need to use the URL of an overview / abstract page etc. I’m not sure why, since this does work in Zotero. More info might be here[13].

2. Currently, this filter only works if you use pandoc-citeproc, because the citations are written directly into the document metadata instead of into a bibtex file. If you want to use natbib or biblatex for citations, this filter currently won’t work. Using citeproc has the disadvantage that it is somewhat less configurable than the "real" LaTeX citation text generators and the CSL language has some limitations. For example, the bibtex "alpha"[14] style sometimes used in Germany can’t be described in CSL.

   To make it work with biblatex, this script would need to write out a *.bib file somewhere temporarily and reference that in the latex code.

3. Some websites just have wrong meta information. For example, citationstyles.org has set "Your Name" as the website author in their Open Graph[15] metadata.

4. Using URLs directly as citekeys (e.g. [@[https://google.com](https://google.com/ "Linkify Plus Plus")] does not work because of pandoc parsing, see this issue[16]. But it does work for ISBNs and DOIs:

   The book [@isbn:978-0374533557, pp. 15-17] is interesting.
   

   See this example[17].

AlexNet[18] first introduced CNNs to the ImageNet challenge. [19]–[21] further improved on the results.

[1] phiresky, “Effortlessly and transparently add correctly styled citations to your markdown paper given only a URL: Phiresky/pandoc-url2cite.” Dec-2019 [Online]. Available: https://github.com/phiresky/pandoc-url2cite. [Accessed: 14-Dec-2019]

[2] Y. Name, “Citation Style Language,” Citation Style Language. [Online]. Available: https://citationstyles.org/. [Accessed: 14-Dec-2019]

[5] “Automatic citation extraction from URLs (draft) - phiresky’s blog,” phiresky.github.io. [Online]. Available: https://phiresky.github.io/blog/2019/pandoc-url2cite/. [Accessed: 14-Dec-2019]

[6] F.-R. Kiko, “How to use Pandoc to produce a research paper,” Opensource.com. [Online]. Available: https://opensource.com/article/18/9/pandoc-research-paper. [Accessed: 14-Dec-2019]

[9] E. Heyns, “Make Zotero effective for us LaTeX holdouts. Contribute to retorquere/zotero-better-bibtex development by creating an account on GitHub.” Dec-2019 [Online]. Available: https://github.com/retorquere/zotero-better-bibtex. [Accessed: 14-Dec-2019]

[11] “A Node.Js-based server to run Zotero translators. Contribute to zotero/translation-server development by creating an account on GitHub.” zotero, Dec-2019 [Online]. Available: https://github.com/zotero/translation-server. [Accessed: 14-Dec-2019]

[15] “Open Graph protocol,” ogp.me. [Online]. Available: http://ogp.me/. [Accessed: 14-Dec-2019]

[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, May 2017 [Online]. Available: http://doi.acm.org/10.1145/3065386. [Accessed: 14-Dec-2019]

[19] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556 [cs], Sep. 2014 [Online]. Available: http://arxiv.org/abs/1409.1556. [Accessed: 14-Dec-2019]

[20] C. Szegedy et al., “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9 [Online]. Available: https://ieeexplore.ieee.org/document/7298594. [Accessed: 14-Dec-2019]

[21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778 [Online]. Available: https://ieeexplore.ieee.org/document/7780459. [Accessed: 14-Dec-2019]
https://phiresky.github.io/blog/2019/pandoc-url2cite/?