DOCX -> MD -> DOCX while keeping Zotero embedded / live citations intact

[the-solipsist]

I'm collaborating with a number of people who use MS Word, with a shared Zotero library. I've requested them to add citations to the DOCX files using Zotero (i.e., live / embedded citations, rather than plain text citations). This DOCX file also has comments and tracked changeds. I can convert this to Markdown using:

pandoc -f docx+citations test.docx --track-changes=all --standalone -t markdown -o test.md

However, when I do so, the id-s for the references end up being numbers, rather than the citekey used in our shared Zotero library.

I've had a look at https://retorque.re/zotero-better-bibtex/exporting/pandoc/ but that seems to require the citekey.

Thus I'm having difficulty figuring out how to export the edited Markdown file back into a DOCX file with the Zotero live citations intact. Any help would be appreciated.

[retorquere]

It doesn't look like the pandoc AST retains all relevant information. If you unzip the docx you'll see that the citations have, ao, the uri and the itemData, where the pandoc AST has only the id; but I don't know where that ID comes from, and I have no idea how to correlate it to the Zotero item. You can see the pandoc AST (which is what is available to pandoc filters) using

pandoc -f docx+citations test.docx --track-changes=all --standalone -t native -o test.ast

[the-solipsist]

The citationID in the XML is "f984sDbg", the
citationItems id is "679", and the
citationsItems uris is "http://zotero.org/groups/[xxxx]/items/FS7WXJRW"

while the id in the Pandoc AST is "679".

If I understood you correctly, if Pandoc includes the citationItems.uris and the citationID, then you'd be able to ensure round-tripping of the Zotero live citations?

---

What's in the XML:

ADDIN ZOTERO_ITEM CSL_CITATION
{"citationID":"f984sDbg","properties":{"formattedCitation":"\\uc0\\u8220{}The
Rule of Law | Daily FT,\\uc0\\u8221{} accessed November 7, 2023,
https://www.ft.lk/columns/The-rule-of-law/4-722553.","plainCitation":"“The
Rule of Law | Daily FT,” accessed November 7, 2023,
https://www.ft.lk/columns/The-rule-of-law/4-722553.","noteIndex":4},"citationItems":[{"id":679,"uris":["http://zotero.org/groups/4962152/items/FS7WXJRW"],"itemData":{"id":679,"type":"webpage","abstract":"As
a country Sri Lanka is currently in a great crisis which cannot be
overcome only by the Government in power; it requires a very strong
and consolidated national effort involving all Opposition parties
and the people in general. Unfortunately both the ruling party and
the Opposition have failed to realise that.\nSri Lanka is a deaf
and dumb country that cannot be awakened by prior warnings. It is
..","language":"English","title":"The rule of law | Daily
FT","URL":"https://www.ft.lk/columns/The-rule-of-law/4-722553","accessed":{"date-parts":[["2023",11,7]]}}}],"schema":"https://github.com/citation-style-language/schema/raw/master/csl-citation.json"}
</w:instrText>

What's in the AST:

                , MetaMap
                    (fromList
                       [ ( "URL"
                         , MetaInlines
                             [ Str
                                 "https://www.ft.lk/columns/The-rule-of-law/4-722553"
                             ]
                         )
                       , ( "abstract"
                         , MetaInlines
                             [ Str "As"
                             , Space
                             , Str "a"
                             , Space
                             , Str "country"
                             , Space
                             , Str "Sri"
                             , Space
                             , Str "Lanka"
                             , Space
                             , Str "is"
                             , Space
                             , Str "currently"
                             , Space
                             , Str "in"
                             , Space
                             , Str "a"
                             , Space
                             , Str "great"
                             , Space
                             , Str "crisis"
                             , Space
                             , Str "which"
                             , Space
                             , Str "cannot"
                             , Space
                             , Str "be"
                             , Space
                             , Str "overcome"
                             , Space
                             , Str "only"
                             , Space
                             , Str "by"
                             , Space
                             , Str "the"
                             , Space
                             , Str "Government"
                             , Space
                             , Str "in"
                             , Space
                             , Str "power;"
                             , Space
                             , Str "it"
                             , Space
                             , Str "requires"
                             , Space
                             , Str "a"
                             , Space
                             , Str "very"
                             , Space
                             , Str "strong"
                             , Space
                             , Str "and"
                             , Space
                             , Str "consolidated"
                             , Space
                             , Str "national"
                             , Space
                             , Str "effort"
                             , Space
                             , Str "involving"
                             , Space
                             , Str "all"
                             , Space
                             , Str "Opposition"
                             , Space
                             , Str "parties"
                             , Space
                             , Str "and"
                             , Space
                             , Str "the"
                             , Space
                             , Str "people"
                             , Space
                             , Str "in"
                             , Space
                             , Str "general."
                             , Space
                             , Str "Unfortunately"
                             , Space
                             , Str "both"
                             , Space
                             , Str "the"
                             , Space
                             , Str "ruling"
                             , Space
                             , Str "party"
                             , Space
                             , Str "and"
                             , Space
                             , Str "the"
                             , Space
                             , Str "Opposition"
                             , Space
                             , Str "have"
                             , Space
                             , Str "failed"
                             , Space
                             , Str "to"
                             , Space
                             , Str "realise"
                             , Space
                             , Str "that."
                             , SoftBreak
                             , Str "Sri"
                             , Space
                             , Str "Lanka"
                             , Space
                             , Str "is"
                             , Space
                             , Str "a"
                             , Space
                             , Str "deaf"
                             , Space
                             , Str "and"
                             , Space
                             , Str "dumb"
                             , Space
                             , Str "country"
                             , Space
                             , Str "that"
                             , Space
                             , Str "cannot"
:                             , Space
                             , Str "be"
                             , Space
                             , Str "awakened"
                             , Space
                             , Str "by"
                             , Space
                             , Str "prior"
                             , Space
                             , Str "warnings."
                             , Space
                             , Str "It"
                             , Space
                             , Str "is"
                             , Space
                             , Str ".."
                             ]
                         )
                       , ( "accessed" , MetaString "2023-11-07" )
                       , ( "id" , MetaString "679" )
                       , ( "language" , MetaInlines [ Str "English" ] )
                       , ( "title"
                         , MetaInlines
                             [ Str "The"
                             , Space
                             , Str "rule"
                             , Space
                             , Str "of"
                             , Space
                             , Str "law"
                             , Space
                             , Str "|"
                             , Space
                             , Str "Daily"
                             , Space
                             , Str "FT"
                             ]
                         )
                       , ( "type" , MetaString "webpage" )
                       ])

[retorquere]

If I understood you correctly, if Pandoc includes the citationItems.uris and the citationID, then you'd be able to ensure round-tripping of the Zotero live citations?

Either one will do, but these are likely Zotero-specific, and the pandoc AST is meant to be generic.

[the-solipsist]

But given that Zotero is used by very many people who use Pandoc for scholarly writing, I think a case can be made to retain extra information in the AST, which specific writers and lua plugins can choose to use or not use. I'll open an issue with Pandoc and check if @jgm is willing to add this to the AST. I feel it would really help. Thanks, Emiliano!

…

On Thu, 2 May 2024 at 14:17, Emiliano Heyns ***@***.***> wrote: If I understood you correctly, if Pandoc includes the citationItems.uris and the citationID, then you'd be able to ensure round-tripping of the Zotero live citations? Either one will do, but these are likely Zotero-specific, and the pandoc AST is meant to be generic. — Reply to this email directly, view it on GitHub <#2862 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXQE6GSR4FNFRDTCOGQL5DZAH4QLAVCNFSM6AAAAABHCFNBFCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TEOJTGA2TQ> . You are receiving this because you authored the thread.Message ID: <retorquere/zotero-better-bibtex/repo-discussions/2862/comments/9293058@ github.com>

[jgm]

Changes to the pandoc AST are big and messy and we avoid them at all costs.

I note that the pandoc data structure contains a citationHash field which was used in the early pandoc-citeproc library but isn't used any more by pandoc or citeproc. In principle one could stuff the URI into this, but it seems...unprincipled.

[jgm]

It just takes them from the citeproc id field embedded in the docx.
If you have access to the bibliography, then you could use a filter to look up the id and substitute the url for the citationId in the pandoc AST.

[retorquere]

The ID is really only meaningful inside the document; zotero uses the uri field inside the embedded CSL-JSON to connect in-doc citations to library items.

But when dumping the AST I see a block starting with

Pandoc
  Meta
    { unMeta =
        fromList
          [ ( "references"

and that has the data of the references -- including the id field and the citation-key field for each. If pandoc would use that citation-key if present, rather than the id, that would get the desired result. If not, I'll have to create a second filter to parse the Meta block, grab the id/citation-key combo, and use that to swap the citationId attribute in Citations.

[retorquere]

@the-solipsist can you open a feature request on https://github.com/jgm/pandoc/issues for taking the citation-key from the citations (if present) rather than the id? I think that would solve the issue.

[retorquere]

(previous message removed -- I'm such an idiot)

@the-solipsist this will do it! If you can verify I will put it up on the BBT site:

local citekeys = {}
function Meta(meta)
  if meta.references ~= nil then
    for i, ref in ipairs(meta.references) do
      if ref.id ~= nil and ref["citation-key"] ~= nil then
        citekeys[ref.id] = pandoc.utils.stringify(ref["citation-key"])
      end
    end
    meta.references = nil
  end

  return meta
end

function Cite(cite)
  for _, item in pairs(cite.citations) do
    local citekey = citekeys[item.id]
    if citekey ~= nil then
      item.id = citekey
    end
  end

  return cite
end

return {
  { Meta = Meta },
  { Cite = Cite },
}

ran as

pandoc -s --lua-filter=docx+citations.lua -t markdown -o reverse.md -f docx+citations docx+citations.docx

[retorquere]

@jgm @the-solipsist this script does what I proposed above pandoc would do natively.

[jgm]

Feel free to put up an issue on pandoc for this. I'll ponder it further but you shouldn't expect anything to change in the near future.

[the-solipsist]

Wow. Thanks, @retorquere. I'm currently at an airport. I'll try out the lua filter once I'm in front of my laptop, and will let you know.

…

On Fri, 3 May, 2024, 12:45 Emiliano Heyns, ***@***.***> wrote: @jgm <https://github.com/jgm> @the-solipsist <https://github.com/the-solipsist> this script does what I proposed above pandoc would do natively. — Reply to this email directly, view it on GitHub <#2862 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXQE6A4SOII24Z75AHOV2LZANTETAVCNFSM6AAAAABHCFNBFCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMBUGM4DO> . You are receiving this because you were mentioned.Message ID: <retorquere/zotero-better-bibtex/repo-discussions/2862/comments/9304387@ github.com>

[the-solipsist]

Hi. I'm sorry for the delay in getting back. That lua filter didn't work for me, unfortunately.

sol@spica ~/d/w/l/d/c/v2 (main)> cat citations.lua 
local citekeys = {}
function Meta(meta)
  if meta.references ~= nil then
    for i, ref in ipairs(meta.references) do
      if ref.id ~= nil and ref["citation-key"] ~= nil then
        citekeys[ref.id] = pandoc.utils.stringify(ref["citation-key"])
      end
    end
    meta.references = nil
  end

  return meta
end

function Cite(cite)
  for _, item in pairs(cite.citations) do
    local citekey = citekeys[item.id]
    if citekey ~= nil then
      item.id = citekey
    end
  end

  return cite
end

return {
  { Meta = Meta },
  { Cite = Cite },
}

Without citations.lua:

sol@spica ~/d/w/l/d/c/v2 (main)> pandoc d4dasia_country-report_lk_v2.docx -f docx+citations --track-changes=all -t markdown --standalone | pee head tail
---
references:
- abstract: At the outset it should be said that there is no question as
    to the need or importance of a data protection legislation. Sri
    Lanka has long needed a comprehensive legislation to protect
    individuals' personal data from breach and unauthorised use.
    However, a reading of the Personal Data Protection 2021, draft in
    its present form, (Draft 7 published in February 2021 in ICTA's
    official website, h ..
  accessed: 2023-11-10

[^112]: [@1050]

[^113]: [@1921]

[^114]: [@1921]

[^115]: [@1048]

[^116]: [@1048]

With citations.lua:

sol@spica ~/d/w/l/d/c/v2 (main)> pandoc d4dasia_country-report_lk_v2.docx --lua-filter citations.lua -f docx+citations --track-changes=all -t markdown --standalone | pee head tail
**Title**

[redacted text]


[^112]: [@1050]

[^113]: [@1921]

[^114]: [@1921]

[^115]: [@1048]

[^116]: [@1048]

So with citations.lua, the csl-yaml references goes away from the yaml frontmatter, and the [@Number] in the footnotes remains the same.

[retorquere]

That's fine.

[retorquere]

I've looked at the documents you sent -- since the docx document you added comments to no longer has Zotero live citations (only the rendered text remains), the filter can't do anything -- it is waiting for Cite nodes to be handed to it by pandoc, which are no longer in the document. The filter is not looking at the rendered text, but is looking specifically for Zotero live citations.

[the-solipsist]

To be clear, it is the pre-comments document with Zotero live citations that I'm trying to run through the lua filter and not the post-comments document without Zotero live citations.

[retorquere]

Oh, you want it resolved on the Zotero item key in the document. Those aren't made available in the AST the Lua filter gets, so that doesn't appear to be possible. It should be possible by eg parsing the document XML using python or somesuch to restore the citation keys as the citation ID, and then it could be put through pandoc.