Add support for publishing org-roam file nodes via :input-source

[nanzhong]

This is an initial attempt at adding support for using weblorg to publish org-roam file nodes.

I'm very new to emacs lisp, so please feel free to point out anything that is not correct, not idiomatic, or if there are better ways to do something.

On a high level this PR introduces 3 things:

1. New input source functions for org-roam that can be used with weblorg-route: weblorg-input-source-org-roam-nodes and weblorg-input-source-org-roam-nodes-agg
2. Support for rendering org id links (by adding a new configuration value to weblorg-site that specifies the default route to use for id links)
3. Change :input-sourceto support both list and function. This is needed becauseweblorg--route-install-template-filters`, which configures the custom link exporters, happens after the route is defined. This means that make use of the link exporters, we need to run the org -> html export after the site/routes are defined.

Each file node is represented as an alist similar to regular org files, however, there is an additional backlinks key that can be used to render backlinks for a given file node.

There is still quite a bit missing such as tests, documentation, etc. but I wanted to get this up to make sure it's going in the right direction before spending time on those pieces.

Example usage

I've been using this for my own org-roam notes, the following are the relevant parts that show how this can be used (I did not include the theme template's but they should be relatively self-explanatory).

...

(use-package org
  :config
  (setq org-src-preserve-indentation t
        org-html-toplevel-hlevel 1
        org-export-with-section-numbers nil
        org-id-locations-file (expand-file-name "./org-id-locations")))

(use-package org-roam
  :after org
  :init
  (setq org-roam-v2-ack t)
  :config
  (setq org-roam-directory (expand-file-name ".")
        org-roam-file-extensions '("org")
        org-roam-db-location (expand-file-name "./org-roam.db")))

...

(let* ((site (weblorg-site
              :base-url weblorg-default-url
              :default-route "nodes"

       ...

       (org-roam-nodes-filter (lambda (node)
                                  (member "publish" (cdr (assoc "tags" node))))))

  (weblorg-route
   :name "index"
   :input-source (lambda () (weblorg-input-source-org-roam-nodes-agg org-roam-nodes-filter))
   :template "index.html"
   :output "output/index.html"
   :url "/"
   :site site)
  (weblorg-route
   :name "nodes"
   :input-source (lambda () (weblorg-input-source-org-roam-nodes org-roam-nodes-filter))
   :template "node.html"
   :output "output/{{ slug }}/index.html"
   :url "/{{ slug }}/"
   :site site)
   
   ...

  (weblorg-export))

[reyman]

Thank you so much to work on that @nanzhong , i'm ready to beta test when it's done :) !

[nanzhong]

@reyman thanks. I've been using this personally for a little while now and it fits my use cases pretty well. Feel free to give it a go, I'd definitely appreciate that! Happy to address any issues you find and incorporate any additional features you would find helpful. I've added an example site that you can take a look at to see how this extension can be used.

@clarete does the direction that this is going look good to you? The last few bits that are missing are tests and updating the existing documentation (adding new docs for the org-roam extension and updating the change to input-source). I'm happy to start working on those if thing look generally good to you. I don't write a lot of elisp so please feel free to review with a fine tooth comb 😄.

Since I make use of this myself, I'm happy to maintain this extension going forward.

[reyman]

@nanzhong I start to config my site using weblorg, your first example is really helpfull, perhaps adding some more complicated filters query like :

• multiple routes for multiple tags
• not operator, for example : get all nodes without "private" tag
• and/or operators, for example : get all node without "private" and "publish"
• etc.

I try to do that by myself too but some more doc with examples of filters could be good for lisp noob like me :)
I have an error when i try the simple example (I just modifying the tag "public" by "PROJECT" and remove call to init because i run this interactively) into doom emacs with emacs native 28 (perhaps linked to this ?) :

Debugger entered--Lisp error: (wrong-type-argument org-roam-node nil)
  signal(wrong-type-argument (org-roam-node nil))
  (or (progn (and (memq (type-of node) cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node)))
  (progn (or (progn (and (memq (type-of node) cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node))) (aref node 16))
  (member "PROJECT" (progn (or (progn (and (memq (type-of node) cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node))) (aref node 16)))
  (lambda (node) (member "PROJECT" (progn (or (progn (and (memq (type-of node) cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node))) (aref node 16))))(nil)
  #f(compiled-function (node) #<bytecode 0x88c8788df02fb77>)(nil)
  #f(compiled-function (elt) #<bytecode -0x2bb1885ff9e7836>)(nil)
  mapcar(#f(compiled-function (elt) #<bytecode -0x2bb1885ff9e7836>) (... ... ... ... ... ... ... ... ... ... ... ... ... ... ...))
  #f(compiled-function #'sequence #<bytecode 0x1843cfe2f231a434>)(#f(compiled-function (elt) #<bytecode -0x2bb1885ff9e7836>) (... ... ... ... ... ... ... ... ... ... ... ... ... ... ...))
  apply(#f(compiled-function #'sequence #<bytecode 0x1843cfe2f231a434>) #f(compiled-function (elt) #<bytecode -0x2bb1885ff9e7836>) (... ... ... ... ... ... ... ... ... ... ... ... ... ... ...) nil)
  seq-map(#f(compiled-function (elt) #<bytecode -0x2bb1885ff9e7836>) (... ... ... ... ... ... ... ... ... ... ... ... ... ... ...))
  seq-filter(#f(compiled-function (node) #<bytecode 0x88c8788df02fb77>) (... ... ... ... ... ... ... ... ... ... ... ... ... ... ...))
  weblorg--input-source-org-roam-nodes((lambda (node) (member "PROJECT" (progn (or (progn (and (memq ... cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node))) (aref node 16)))) nil nil)
  weblorg-input-source-org-roam-nodes-agg((lambda (node) (member "PROJECT" (progn (or (progn (and (memq ... cl-struct-org-roam-node-tags) t)) (signal 'wrong-type-argument (list 'org-roam-node node))) (aref node 16)))))
  (lambda nil (weblorg-input-source-org-roam-nodes-agg org-roam-nodes-filter))()
  weblorg-export-templates(#<hash-table eql 16/65 0x157ce7c51eb3>)
  #f(compiled-function (_ route) #<bytecode 0x6146bc495f8b951>)("index" #<hash-table eql 16/65 0x157ce7c51eb3>)
  maphash(#f(compiled-function (_ route) #<bytecode 0x6146bc495f8b951>) #<hash-table equal 2/65 0x157ce95dadbd>)
  #f(compiled-function (_ site) #<bytecode 0x121a1cee697043aa>)("http://localhost:8000" #<hash-table eql 6/6 0x157ce95dad29>)
  maphash(#f(compiled-function (_ site) #<bytecode 0x121a1cee697043aa>) #<hash-table equal 1/65 0x157ce933cea1>)
  weblorg-export()
  (let* ((site (weblorg-site :default-route "org-nodes" :theme nil)) (org-roam-nodes-filter #'(lambda (node) (member "PROJECT" (progn (or ... ...) (aref node 16)))))) (weblorg-route :name "index" :input-source #'(lambda nil (weblorg-input-source-org-roam-nodes-agg org-roam-nodes-filter)) :template "index.html" :output "output/index.html" :url "/" :site site) (weblorg-route :name "org-nodes" :input-source #'(lambda nil (weblorg-input-source-org-roam-nodes org-roam-nodes-filter)) :template "org-node.html" :output "output/{{ slug }}/index.html" :url "/{{ slug }}/" :site site) (setq debug-on-error t) (weblorg-export))
  eval-buffer()  ; Reading at buffer position 875
  funcall-interactively(eval-buffer)
  command-execute(eval-buffer record)
  execute-extended-command(nil "eval-buffer" nil)
  funcall-interactively(execute-extended-command nil "eval-buffer" nil)
  command-execute(execute-extended-command)

I try different things, verifying that tags are ok using that, but i don't found a solution.

(defun org-roam-filter-by-tag (tag-name)
  (lambda (node)
    (member tag-name (org-roam-node-tags node))))

(defun org-roam-list-notes-by-tag (tag-name)
  (mapcar #'org-roam-node-file
          (seq-filter
           (reyman/org-roam-filter-by-tag tag-name)
           (org-roam-node-list))))

(org-roam-list-notes-by-tag "PROJECT")

I use :

Org mode version 9.6 (9.6-??-0c9b30e @ /home/reyman/.emacs.d/.local/straight/build-28.0.90/org/)
Org roam version "679ef6e" (latest)