Sort entries by due date when using org-ql-block

[jakejx]

Hello, thank you for this wonderful package, it makes my custom agendas useable again!

I am in the process of porting some of my custom agenda commands I was wondering whether it was possible to sort my results based on the deadline. I tried using the conventional way of setting org-agenda-sorting-strategy, but I don't think it works. Here is what my agenda command looks like

(org-ql-block '(and (tags "test")
                    (not (tags "CANCELLED" "HOLD"))
                    (not (scheduled))
                    (or (or (todo "NEXT") (todo "PROG"))
                        (deadline 3)))
              ((org-ql-block-header "TEST")
                (org-agenda-sorting-strategy '(deadline-up))))

Is setting the sorting strategy like this supported? Thanks!

[alphapapa]

Thanks for the kind words. I'm glad it's helpful to you.

Actually, there is no way to do that at the moment. That's an oversight on my part. I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search. Thanks for letting me know!

[yantar92]

Seconding the request

[jakejx]

Thanks for the kind words. I'm glad it's helpful to you.

Actually, there is no way to do that at the moment. That's an oversight on my part. I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search. Thanks for letting me know!

No apologies needed! I will try to take a look at the code too 😄

[akirak]

I'll need to add additional arguments to the block function so :sort can be specified, like in org-ql-search.

Or maybe you can support the feature by adding a variable for a sorting strategy, like org-ql-block-header does for the header. I personally don't want to contain something other than a predicate in the second argument, so I prefer this way.

[alphapapa]

@akirak That's a good point.

What do the rest of you think?

[yantar92]

What do the rest of you think?

I feel that it would be better to stick to built-in org-mode functions as much as possible. Otherwise, you may end up reimplementing the whole org-agenda.el from scratch. I believe that only the most performance-critical functions from org-agenda need to be reimplemented.

[alphapapa]

@yantar92 The question here is only how to present the sorting argument in the block definition, not the implementation.

In fact, org-ql is essentially a reimplementation of the agenda--that's the point, because org-agenda.el is very difficult to understand and modify, and nearly impossible to extend.

[jakejx]

I am ambivalent to either solution. But of course whichever method is easier to leverage upon the built in org mode functionality should be preferred, to make implementation easier.

[alphapapa]

@jakejx Sorting is already implemented in org-ql. org-agenda sorting functions are not used. (If you're curious as to why, look at the code for the function org-entries-lessp. @akirak you might find it amusing.)

The question here is whether to pass the sort argument like this, which is consistent with org-ql functions:

(setq org-agenda-custom-commands
      '(("ces" "Custom: Agenda and Emacs SOMEDAY [#A] items"
         ((org-ql-block ('(and (todo "SOMEDAY")
                               (tags "Emacs")
                               (priority "A"))
                         :sort '(date priority todo)))
          (agenda)))))

or like this, which is more like org-agenda block settings:

(setq org-agenda-custom-commands
      '(("ces" "Custom: Agenda and Emacs SOMEDAY [#A] items"
         ((org-ql-block '(and (todo "SOMEDAY")
                              (tags "Emacs")
                              (priority "A"))
                        ((org-ql-block-sort '(date priority todo))))
          (agenda)))))

I'm generally in favor of the first example, because it's consistent with org-ql's API, and it doesn't require an extra binding form in the block, which makes the list structure much simpler.

[akirak]

@alphapapa

(org-ql-block ('(and (todo "SOMEDAY")
                               (tags "Emacs")
                               (priority "A"))
                         :sort '(date priority todo)))

This looks better than I expected. Actually, I now even prefer it over the other, because it allows easy migration to/from normal org-ql forms and requires less memorization of variable names.

If you're curious as to why, look at the code for the function org-entries-lessp. @akirak you might find it amusing.

Thanks, I'll take a look at it.

@yantar92

I feel that it would be better to stick to built-in org-mode functions as much as possible. Otherwise, you may end up reimplementing the whole org-agenda.el from scratch.

org-ql started out as a library for org-agenda-ng, which was an experiment by alphapapa to improve upon the API of org-agenda. org-agenda-ng was merged into org-ql. As he develops org-ql, it is becoming a practical alternative to org-agenda.

[yantar92]

In a view on org-ql as a replacement to org-agenda, it indeed makes sense to stick to org-ql syntax rather than retaining the old org-agenda syntax. Mixing up the syntax will make things confusing.

Also, replacing org-ql-block-header with :header will make sense then.

[alphapapa]

I looked into making sorting in org-ql-block blocks sort according to org-agenda-sorting-strategy, but it doesn't appear to be possible because of the difference in how Org Agenda and org-ql are implemented.

Org Agenda performs sorting based on a type text property which is set on entry strings depending on why the entry was collected. For example, when preparing a daily/weekly agenda, an entry selected because of its deadline has the property set to "deadline", while one selected because it was scheduled in the past has "past-scheduled". This is possible because entries are selected in multiple passes through each buffer, e.g. once to collect deadline items, once to get scheduled items, etc. But org-ql selects entries based on a given query in a single pass through a buffer, so it's not possible to know the Org Agenda-equivalent of the "reason" that an entry is collected, so the corresponding type property can't be set, so there's no such property for org-entries-lessp to sort on.

Anyway, I found the way to make org-ql's sorting work within an org-ql-block block.

However, I realized that changing the way org-ql-block's arguments are passed will break the configs of users who are running master (which is 99.9% of users, of course, since virtually no one uses MELPA Stable). So I think this change should be postponed to 0.5, so at least we can have a changelog entry, and maybe some code to warn about the changing argument format. (Not that many people read the changelog, anyway, but at least it will be there if someone notices the breakage and needs to find out why.)

So I've pushed this WIP branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79 for 0.5.

[bitclick]

this could be another bug, but for me it serves as a workaround:

org-ql-block applies the sorting-strategy of the preceding normal agenda-block.

i have a dummy block before the org-ql-block and apply the sorting strategy there. the results from the org-ql-block are sorted like i want them to:

...
(todo "nothing here"
      (
       (org-agenda-overriding-header "")
       (org-agenda-sorting-strategy '(category-keep))
       )
  )
(org-ql-block  ...) 
...

maybe this is related to #121, in the sense that there is some cleanup missing when initializing the agenda-buffer for org-ql.

anyways, now i can sort the way i like :-)

[alphapapa]

@bitclick I'm afraid that's not the case. As I explained in #79 (comment), org-ql blocks are not compatible with org-agenda sorting.

You can test this for yourself as follows: In a clean Emacs instance:

1. Create file test79.org with these contents:

* Heading 1
<2020-07-23 Thu 09:00>

* Heading 2
<2020-07-24 Thu 09:00>

2. Add that file to the agenda list.
3. Evaluate this expression:

(setq org-agenda-custom-commands
      '(("A" "Today"
	 ((todo "nothing here"
		((org-agenda-sorting-strategy '(timestamp-down))))
	 (org-ql-block '(ts)
		       ((org-ql-block-header "TEST")))))))

4. Display that custom agenda.
5. Note the order of the headings: 1 followed by 2.
6. Change the sorting strategy to timestamp-up.
7. Display that custom agenda anew.
8. Note that the order of the headings is unchanged.

[YujiShen]

New to org-ql and org-super-agenda. I am trying to sort items by priority within a group defined by org-ql-block in org-agenda-custom-commands. Just saw this is still open. Does it mean I can not sort items within group of org-super-agenda with org-ql?

I saw the doc of org-super-agenda said:

The order of items may not be preserved after grouping due to the implementation’s using hash tables. Future versions may address this shortcoming.

But I also saw a Reddit thread said:

Sorting is controlled by Org Agenda settings (or org-ql, if you're using it).

So I am very confused: Is it possible to sort within a group of org-super-agenda (using org-ql or not)?

[xanalogica]

Anyway, I found the way to make org-ql's sorting work within an org-ql-block block.

So I've pushed this WIP branch: https://github.com/alphapapa/org-ql/tree/wip/issue-79 for 0.5.

Thanks for this work. I've been trying it out and found it tricky to properly declare org-agenda-custom-command so I'm pasting what I got working for others who make wander by. Pardon the spread-out syntax -- I had to do that to figure out what I kept doing wrong -- which was an incorrect combination of quoting.

  ("c" "<c>aptured things"        ;; [1] key  [2] description
    org-ql-block                  ;; [3] function that select items
    '(                            ;; [4] match/query expression
      (property "CAPTURED")       ;;   arg1: org-query expression
      :sort (priority date)       ;;   keyword arg
      :header "xyzzy"             ;;   keyword arg
     )
    (                             ;; [5] list of extra settings
    )
    ()                            ;; [6] list of files to export to
  ) ;; end of agenda entry definition

[alphapapa]

Since this involves changing the format, what should probably be done is:

1. In 0.6, warn about changing argument format when the old format is used.
2. In 0.7, only accept the new format. (Since this change will be made on master, it should probably be delayed for at least a few weeks after 0.6 is released.)

[alphapapa]

Retargeting this for 0.7. 0.6 has been delayed for too long.

[alphapapa]

Just curious, is there anything preventing this from being merged to master?

AFAICT I've been waiting for someone to confirm that it still works. At this point I should rebase it again first.

[alphapapa]

Please see #459. My time is limited, so if someone else could test it and confirm that it works as expected, I'll document the changes and merge it.

[lyndhurst]

If trying out that branch and reporting back can help, I would gladly do it as soon as you get the time to rebase.
I subscribed to this thread a little while ago because I would love to see this feature added.

[alphapapa]

@lyndhurst The PR #459 is rebased. Feel free to test it. Thanks.

[lyndhurst]

Thanks for the link, it is night time in my country, I will report back first thing in the morning.

[lyndhurst]

I had little time this morning, so I only installed the wip/issue-79 branch and tested my org-ql blocks as is. I had completely unexpected results. I need to test again with more method before sharing anything reliable. I will get to it in a couple of hours, and report back. Sorry for the delay.

[lyndhurst]

@alphapapa , I did some more testing, removing all the org-ql customizations and overrides I have in my config. And I get the same unexpected outcome.

I have not tried any :sort criterias yet, but for the little testing I did, using the wip/issue-79 branch, using and and/or or filters do not work anymore.

I created the following file:

#+title: Test - Org-ql Sort 
#+description: Testing org-ql sort feature branch

* TODO Unscheduled todo
* TODO Scheduled todo
SCHEDULED: <2024-09-02 Mon>
* NEXT Schedduled next
SCHEDULED: <2024-09-04 Wed>

And used this config:

(setq org-agenda-files '("~/.local/share/org/tasks/test_test_testing_sort_feature.org"))
(setq org-agenda-custom-commands
      '(("z" "Tests"
	 (
          
            (org-ql-block '(scheduled)
                          (
                           (org-ql-block-header "Test SCHED")
                           (org-super-agenda-groups nil)
                           ))
          
            (org-ql-block '(or
                            (todo "TODO")
                            (scheduled)
                            )
                          (
                           (org-ql-block-header "Test OR")
                           (org-super-agenda-groups nil)
                           ))

            (org-ql-block '(and
                            (todo "TODO")
                            (scheduled)
                            )
                          (
                           (org-ql-block-header "Test AND")
                           (org-super-agenda-groups nil)
                           ))

          ))))

Sorry for the unreadable file naming and ugly formatting, but, I used templates already set up in my config, and the formatting makes it easier for me to do a lot of changes quickly.

Anyway, as you can see in the screenshot, and seems to show every item without filtering, or filters out everything, only the 'Test SCHED' shows expected results. This outcome is consistent with the first results I had using my own org-ql-block views that are working fine on master.

Finally I have to specify that I am a Doom Emacs user and I switched to Emacs around 6 months ago, so I might be doing something wrong. Please let me know if I can do any more testing, or if you need additional info about the versions I have installed and so on..

[alphapapa]

@lyndhurst It looks like you're using the old argument style for org-ql-block. The reason this PR wasn't merged already is that it changes that. You need to use the new format. See #79 (comment)

[lyndhurst]

Thank you @alphapapa for the link. I had seen it before and wondered if there was a syntax mistake on my part.

To be honest it took me a little while to get it to work. I think the main issue was that no errors are thrown when I make syntax mistakes, and I spent my time thinking I was stupid because the outcome did not make sense while actually I was stupid because I was not using check-parens enough :)

Anyway, I added a little data to the test file to have more sorting material:

#+title: Test - Org-ql Sort 
#+description: Testing org-ql sort feature branch

* TODO [#B] Unscheduled medium
* TODO [#A] Unscheduled high
* TODO [#B] Scheduled todo
SCHEDULED: <2024-09-02 Mon>
* NEXT [#A] Scheduled next
SCHEDULED: <2024-09-04 Wed>

I managed to get a few blocks going with the following config:

(setq org-agenda-files '("~/.local/share/org/tasks/test_test_testing_sort_feature.org"))
(setq org-agenda-custom-commands
      '(("z" "Tests"
	 (
          
          (org-ql-block '(scheduled
                         :sort (date)
                         :header "Test Date Sort")
                        )
          
          (org-ql-block '(scheduled
                         :sort (priority)
                         :header "Test Priority Sort")
                        )
          
          (org-ql-block '((or
                           (todo "TODO")
                           (scheduled))
                         :sort (date priority)
                         :header "Test OR Double Sort")
                        )
          
          (org-ql-block '((or
                           (todo "TODO")
                           (scheduled))
                         :sort (priority date)
                         :header "Test OR Inverted Sort")
                        )
          
          (org-ql-block '((and
                           (todo "TODO")
                           (scheduled))
                         :sort (priority)
                         :header "Test AND Filter")
                        )
          
          )
         )))

It worked as I expected. Except maybe for the multiple sort criterias test blocks, it works fine, but the criterias have to be specified in the opposite order I would have expected:

The last block is cut off, but not really interesting; it is a simple (and) operator test to make sure I can make it work later :)

I also tried adding more options the the blocks:

          
          (org-ql-block '((and
                           (todo "TODO")
                           (scheduled))
                         :sort (priority)
                         :header "Test AND Filter")
                        (org-super-agenda-groups nil))

It did not throw any error or unexpected results, but then I realized this testing file does not have anything to test groups on, so I am still unsure how to make that part of the new syntax work.

I think this test shows that the sort functions, and basic functionalities work on my side. I would like to try with my regular blocks which are much more involved in terms of query filters, but it will take me a little more time to adjust them to the new syntax. I could do it tomorrow and let you know if it is of any interest.

Please let me know if there is anything else I can do to help.

[lyndhurst]

I was wrong about not having anything to test groups on: this works:

          (org-ql-block '((or
                           (todo "TODO")
                           (scheduled))
                         :sort (priority date)
                         :header "Test OR Inverted Sort")
                         ((org-super-agenda-groups '((:auto-planning))))
                        )

And the blue header 'Test' that appears in every one of the screenshot's blocks are actually coming from my personal super-agenda-groups settings.

[alphapapa]

@lyndhurst I'm afraid that some of those blocks still have incorrect syntax. Here is what they should look like:

(let ((org-agenda-files '("/tmp/test.org"))
      (org-agenda-custom-commands
       '(("z"
          "Tests"
          ((org-ql-block '((scheduled)
                           :sort (date)
                           :header "Test Date Sort"))
           (org-ql-block '((scheduled)
                           :sort (priority)
                           :header "Test Priority Sort"))
           (org-ql-block '((or (todo "TODO") (scheduled))
                           :sort (date priority)
                           :header "Test OR Double Sort"))
           (org-ql-block '((or (todo "TODO") (scheduled))
                           :sort (priority date)
                           :header "Test OR Inverted Sort"))
           (org-ql-block '((and (todo "TODO") (scheduled))
                           :sort (priority)
                           :header "Test AND Filter")))))))
  (org-agenda nil "z"))

Note that that form can be evaluated as-is to test it, without changing the global bindings of those variables. I recommend using this pattern to test things with. Also, I recommend using aggressive-indent-mode for editing Elisp code, and avoiding hanging parens on their own lines. That way your indentation and formatting will be consistent and easily readable.

Except maybe for the multiple sort criterias test blocks, it works fine, but the criterias have to be specified in the opposite order I would have expected:

Please see the note in the changelog about the reverse sort method. Sorting is specified as a list of imperatives, i.e. as if you told org-ql to first sort by method A, then by B--each method sorts all of the items, so the last one given will effectively have priority over others. The reverse method can be used anywhere in the list of sort methods to reverse the items at that place in the application of sorting methods. (As you can see, explaining how this works isn't easy, especially when trying to do so in less-technical terms that might be understood by Org users who might not be programmers.)

[lyndhurst]

Thanks for the patience, I am not able to test further tonight, but the tests worked well in spite of the incorrect syntax. I will have another look tomorrow anyway.

Sorry again about the formatting, I think it is the modal editing from evil-mode, but when testing and moving things around a lot I still have trouble with Elisp. I could format before posting though, that is a bit lazy on my part.

For the last part on reverse, I might have chosen an ambiguous term to name the test block. I was not refering to the reverse method used in programming to modify objects like arrays. If you look at the two 'OR' test blocks, one sorts using (date priority) and the second uses (priority date). I was just looking for way to demonstrate that sorting works on the test file.

I may have to look at it again, because it took me a lot of error and trials to get the syntax to work (even if it is still off ;)), but I had the feeling the order I had to list the criterias in was the opposite of the one I would use in a sql GROUP BY for example. But I might be wrong, I will check again when trying to understand the correct syntax, and if it changes the results I get.

Thanks again for the tips.

[alphapapa]

Thanks for the patience, I am not able to test further tonight, but the tests worked well in spite of the incorrect syntax. I will have another look tomorrow anyway.

No problem. There's no rush here.

Sorry again about the formatting, I think it is the modal editing from evil-mode, but when testing and moving things around a lot I still have trouble with Elisp. I could format before posting though, that is a bit lazy on my part.

No problem. I'm just sharing what I've learned.

For the last part on reverse, I might have chosen an ambiguous term to name the test block. I was not refering to the reverse method used in programming to modify objects like arrays. If you look at the two 'OR' test blocks, one sorts using (date priority) and the second uses (priority date). I was just looking for way to demonstrate that sorting works on the test file.

Yes, I understood what you meant. What I wrote was an attempt to explain how that works.

I may have to look at it again, because it took me a lot of error and trials to get the syntax to work (even if it is still off ;)), but I had the feeling the order I had to list the criterias in was the opposite of the one I would use in a sql GROUP BY for example. But I might be wrong, I will check again when trying to understand the correct syntax, and if it changes the results I get.

You're right that it doesn't work exactly like SQL. It's actually more powerful, because you could write something like :sort (date reverse priority), which would sort by date, then reverse the order, then sort by priority; IOW it's a list of sort operations which are applied in that order. Whether it's the best API is another matter, but there's often not a clear answer; there is somewhat of a mix of declarative and imperative styles involved.

Thanks again for the tips.

I appreciate your help with testing these features.

[lyndhurst]

I took some time today to go over your last message with fresh rested eyes.

Also, I recommend using aggressive-indent-mode for editing Elisp code, and avoiding hanging parens on their own lines. That way your indentation and formatting will be consistent and easily readable.

I just installed it, so I did not have time to write code yet, but, just playing around with it while testing more sorting blocks, it seems really fun to work with; maybe less select/cut/paste for me in the near future :)

Please see the note in the changelog about the reverse sort method. Sorting is specified as a list of imperatives, i.e. as if you told org-ql to first sort by method A, then by B--each method sorts all of the items, so the last one given will effectively have priority over others. The reverse method can be used anywhere in the list of sort methods to reverse the items at that place in the application of sorting methods.

I took the time to read your explanations more carefully, and went to the changelog too. The explanations are clear enough, it does require some focus though. I tinkered with my test file, and the reverse method, it seems like a very powerful tool giving the user a lot of control over the way their lists are displayed. I did not understand yesterday the fact that each element could be reversed independently within the sort method.

(As you can see, explaining how this works isn't easy, especially when trying to do so in less-technical terms that might be understood by Org users who might not be programmers.)

Yes, it might be challenging to explain that part to impatient readers (like me yesterday), or non-technical users. I would say simple examples to get the basic syntax and the general idea might be an interesting approach to get one started. Syntax can be a tough one as well at first, without your example right above, I might still be trying to make it work with some :reverse date or (reverse date) attempts :)

I am especially glad I went back over that part, this is valuable information. Thanks for that one as well.

Anyway, I am not going to pollute the thread any more than necessary. I ran a few tests again, and everything works as expected now even reverse sorting that I tried in some of my 'OR' test blocks. I can post more snippets and screenshots, but I am not sure it is necessary.

Thanks for taking the time to show me the correct syntax, I was not too far off, I guess that is why my test blocks still worked yesterday. Let's say that org-ql-block is a forgiving function :) Also, having the test snippet wrapped in a let method and simply trying it from a code block makes it incredibly simpler. You have no idea how many times I had to recompile my config to get those blocks working 👍

When I first volunteered to do some testing, I had no idea what you needed from the tests, so please let me know if I can add anything else, especially now that I am all setup !

For example I was wondering, if it would help to keep using this branch in my everyday workflow see if anything unexpected occurs, or if that's it for now and I can switch back to master.

[proudot]

I could test it on my daily driver files and it worked flawlessly. Here is my config for reference:

  ("o" "overview"
	   (
	    (org-ql-block '((path "/inbox\.org")
			    :header "Laptop Inbox"))

	    (org-ql-block '((and
			     (not (todo "WAIT"))
			     (or (todo) (and (done) (closed :on today)))
			     (path "next.org")
			     (tags "effort")
			     (level "2"))
			  :sort (priority)
			  :header "Selected efforts"
			  ))
	    (org-ql-block '((and (not (done))
				(planning t)
				(parent (regexp "Tickler")))
			  :sort (date)
			  :header "Tickler"))
	    (agenda)
	    )
	   )

Is that helpful ?

[alphapapa]

@proudot Thanks, that's good to know.

[proudot]

@alphapapa this is very useful to my workflow, so I am happy to help debugging this extensively. Can you explain what is needed to pass the bar for merging ?

[alphapapa]

@alphapapa this is very useful to my workflow, so I am happy to help debugging this extensively. Can you explain what is needed to pass the bar for merging ?

Whenever I have time to finish final tidying up and such. There are a number of PRs on my various projects, and I get to them when I can.

[proudot]

Ho great ! I thought you needed more feedback before you consider it mature enough. My bad. Happy to help debugging stuff when needed.