Always set/save options when exiting transient

[karthink]

I use a Transient menu for configuring options provided by gptel, an LLM interaction package. One request I get quite often from users [1, 2] is to automatically persist all options (transient switches, flags, options etc) between invocations of the menu. Right now only the menu items backed by the transient-lisp-variable class (or its child classes I define) are automatically persistent.

Essentially, they want transient-set/C-x s to be run automatically whenever the menu exits. Is there some way to do this in Transient? Here are a couple of things I've tried:

1. Add (transient-set) inside all suffixes: This has two downsides. (i) Some of the suffixes are regular elisp functions that can be called from outside the transient menu as well, I have to guard against that, and (ii) (transient-set) does not run when they quit the menu, with C-g or q. So they can't invoke the menu to set some options and quit without calling a suffix.

2. Try running (transient-set) in transient-exit-hook (before removing it from the hook). This doesn't work since the hook runs after the transient has already exited. Specifically, this hook is called after (setq transient-current-prefix nil) runs, so it fails.

Is there a way to do this using Transient? My apologies if this is covered in another issue or the manual -- I tried searching for a few different things. I imagine that adding a transient-pre-exit-hook should make this possible, but maybe there's a way to do it after the transient exits as well, since the value of gptel-menu (the transient prefix in question) is stored somewhere?

[tarsius]

I think Transient should provide a mechanism to accomplish this. Adding such functionality out-of-the-box, will take some time, not so much because it is hard to implement, but to allow for the realization, that the first impulse maybe wasn't all that great after all.

There are also many open questions. For example, should we allow per-package and/or per-menu customization? Should individual prefixes be able to override the user preference, if it would be "wrong" (and how to avoid an arms race)? Should saving become a task to be handled by the "pre commands" (I don't think that is feasible, but thinking about it would still be beneficial)? How does this interact with menus that implement their own "automatic persistence" (e.g., magit-diff, which uses local persistence tied to a buffer). ...

So for the time being, you'll have to implement a good enough approach yourself.

... I've tried:

1. Add (transient-set) inside all suffixes: This has two downsides. (i) Some
   of the suffixes are regular elisp functions that can be called from outside
   the transient menu as well, I have to guard against that,

It's a major design goal to be able to use "regular commands", without them having to be modified, so this indeed is undesirable. That being said, if a command should consume the transient arguments, then it must use something like transient-args.

I am wondering (and wouldn't be able to tell, without looking at it in detail), whether commands that don't even consume arguments, should be able to remember arguments. The initial response (karthink/gptel#94 (comment)) to the request to "always save", seems to have been "make the do-it command also save", which is in line with that reasoning.

But by your account that's what users are asking for; "always" save, for "predictability". One aspect I'll have to think about more, is that one persons "consistency" is another's "does completely useless things". I think there is value in providing the "consistent" behavior as an option, but it shouldn't come at the cost of not being able to do the "right (IMO)" thing.

and (ii) (transient-set) does not run when they quit the menu, with C-g
or q. So they can't invoke the menu to set some options and quit without
calling a suffix.

That's a feature, isn't it? If the abort command no longer aborts, but instead saves, then how do you abort, if you really want to abort (e.g., because you changed some arguments, and realize that you have made some mistakes, and want to discard them to go back to what you had before)?

2. Try running (transient-set) in transient-exit-hook (before
   removing it from the hook). This doesn't work since the hook runs
   after the transient has already exited. Specifically, this hook is
   called after (setq transient-current-prefix nil) runs, so it
   fails.

I'll probably add transient-before-exit-hook (and rename transient-exit-hook to transient-after-exit-hook. Since I haven't done that (or something similar) yet, you'll have to use an advice.

(define-advice transient--post-exit (:before (&optional command) gptel-persist)
  (let ((command (or command this-command)))
    (when (and gptel-persist-transient
               (string-prefix-p "gptel-" (symbol-name command)))
      (transient-save))))

This should give you the general idea, but it will require tweaking.

Quoting from karthink/gptel#327 (where you quote from karthink/gptel#291 (comment) ;)

Yes, I realize that I can use C-x s to save the options, but I still find it jarring that gptel forgets some things if I don't do that. It is especially visible if I use I or J to inspect the generated query — I check the query, decide that it's OK, press q to get rid of the query buffer, and the next time I bring up gptel, the setup is different.

Questions:

1. I don't understand the "jarring" bit.

Looking at the screenshot at https://github.com/karthink/gptel/wiki#save-transient-flags, I get the impression, that the jarring bit could be, that you did not manage setting expectations well.

In Magit it is quit obvious which commands set a variable and which set a command line argument. Due to past experience setting variables in configuration files and using argument on the command line, I would think, it is quite intuitive that they behave differently in Transient as well.

But that assumes that one is able to tell the difference. I am not saying you should rush to achieve that, (e.g., by prefixing all the argument keys with -). Maybe we will enable "automatic saving for all", in which case there (probably) wouldn't be much value in it anymore. There could also be other approaches to it (like the exit behavior is visualized using colors), though those could be less intuitive.

So you see, it is quite complex, so for now, play with the given advice function, maybe tweak it a bit, and post the result on the wiki.

[tarsius]

Another place where this could be implemented is transient--history-push. This does remember the used (or effective but ignored) value, it just doesn't save it as the value.

As it is, users can already get to such (automatically) saved values, using C-M-p/C-M-n from a transient. The latest such value just isn't automatically put into effect, when re-entering the transient. Doing that would be yet another potential implementation strategy!

(And maybe that's all very much in line with how LLM themselves work. There's a difference between training and using. Contrary to what users might expect, they don't learn from the interactions you are having with them. If you want to to "recall" past conversations, you have to (explicitly, I assume) provide that context, and there's a limit to how much context you can provide. 😁.)

[karthink]

@tarsius thank you for the detailed, informative and thoughtful response!

---

Persisting transient options

I tried both methods, and while neither one is ideal, I can work with these until you decide if you want to add a Transient feature for it.

gptel-menu-persist is a flag that controls whether options are persistent.

Method 0: Save explicitly

That's a feature, isn't it? If the abort command no longer aborts, but instead saves, then how do you abort, if you really want to abort (e.g., because you changed some arguments, and realize that you have made some mistakes, and want to discard them to go back to what you had before)?

I hadn't considered that. I think I can provide a gptel-transient-save-and-quit command, assigned to q, that is similar to transient-quit-one (C-g), except that it runs transient-set first. Now C-g can continue to follow the "abort" semantics.

This suffix will have to be shown in the menu, unfortunately, as it might be silently rebound to Q or M-q if transient-bind-q-to-quit has run, and the user will not be informed of this change.

Method 1: Advise transient--post-exit

Implementation:

(define-advice transient--post-exit (:before (&optional command) gptel-persist)
  (let ((command (or command this-command)))
    (when (and gptel-menu-persist
               (string-prefix-p "gptel-" (symbol-name command)))
      (transient-set))))

It turns out some of my suffixes are lambdas, so the string-prefix-p clause doesn't quite work. But I should be able to adapt it with some experimentation.

Having gptel advise an internal Transient function seems like a bit of an overreach though. I'll also have to add a gptel-unload-function to remove this advice so unload-feature can work correctly.

Method 2: Load previous settings from transient--history

Implementation:

(transient-define-prefix gptel-menu ()
  "Menu for gptel"
  ;; Rest of spec for menu
  (interactive)
  (transient-setup 'gptel-menu)
  (when gptel-menu-persist
    (transient-history-prev)
    (transient--redisplay)))

Compared to method 1, this version is very clean -- no advice needed. However it doesn't play well with transient-set/C-x s. If the user invokes transient-set manually, then they expect the setting to persist. But the next time they open the menu, it switches to the previous history item. There's no easy way to tell if a history item was set/saved explicitly.

---

I'll reply to your other points in the next comment.

[karthink]

There are also many open questions. For example, should we allow per-package and/or per-menu customization? Should individual prefixes be able to override the user preference, if it would be "wrong" (and how to avoid an arms race)? Should saving become a task to be handled by the "pre commands" (I don't think that is feasible, but thinking about it would still be beneficial)? How does this interact with menus that implement their own "automatic persistence" (e.g., magit-diff, which uses local persistence tied to a buffer). ...

If your question is about enabling per-package/per-menu customization of all Transient options, then I'm not sure. This is a whole can of worms!

But if you meant customization of auto-saving options specifically, the simplest solution would be to follow Transient's lead in other configuration settings -- transient-show-popup, transient-display-buffer-action and transient-enable-popup-navigation are all global settings, so transient-auto-save-options could be global too. When it's disabled, the per-transient behavior (like with magit-diff) can apply. Just a thought.

But by your account that's what users are asking for; "always" save, for "predictability". One aspect I'll have to think about more, is that one persons "consistency" is another's "does completely useless things". I think there is value in providing the "consistent" behavior as an option, but it shouldn't come at the cost of not being able to do the "right (IMO)" thing.

I don't follow. If option persistence is offered as an option (and disabled by default), then everyone can make Transient do the "right" thing for them, can't they? This is excepting the granularity issue, where switching this on for all Transients (or none) may not make sense.

I'll probably add transient-before-exit-hook (and rename transient-exit-hook to transient-after-exit-hook. Since I haven't done that (or something similar) yet, you'll have to use an advice.

That's great! No pressure though, I understand that changes made in a hurry exact a heavy price over time. I will use a workaround for now.

Looking at the screenshot at https://github.com/karthink/gptel/wiki#save-transient-flags, I get the impression, that the jarring bit could be, that you did not manage setting expectations well.

Yeah, there's no analog to command line vs config options in gptel-menu. The model parameters can be construed as either, and the options to the right of the menu (that redirect LLM input and output to other buffers/the kill ring) are analogous to shell pipes, not options. I haven't found a way to signal persistence from the design of the menu.

Maybe we will enable "automatic saving for all", in which case there (probably) wouldn't be much value in it anymore. There could also be other approaches to it (like the exit behavior is visualized using colors), though those could be less intuitive.

The screenshot on the wiki is out of date, this is what it looks like:

Right now all the persistent parameters in gptel-menu (which are instances of transient-lisp-variable or similar) are prefixed by -, and all suffixes are uppercase letters. The = command to switch between setting things globally and buffer-locally serves as a kind of hint that the parameters below it will be set in a persistent way, but it's not obvious.

(And maybe that's all very much in line with how LLM themselves work. There's a difference between training and using. Contrary to what users might expect, they don't learn from the interactions you are having with them. If you want to to "recall" past conversations, you have to (explicitly, I assume) provide that context, and there's a limit to how much context you can provide. 😁.)

Definitely some irony to be mined there. 🙂

Reflecting the behavior of the LLMs in the interface used to access them reminds me of when LLM APIs were first made available last year. There were many simple LLM clients announcing that they were dogfood-ed -- an LLM client written by an LLM! How novel. It looks like this novelty wore off as people quickly reached the limit of this approach.

[tarsius]

Thanks for the detailed reply. I can't work on this now, but the provided information should help a lot getting up to speed, when I return.

[psionic-k]

The pre-command (or another slot) is definitely a good level of granularity of configuration. Not all values provide a good user experience when they are remembered.

Tangentially, I recently I used the transient--do-call pre-command on an infix to just communicate to :inapt-if predicates that needed to see the updated value of the prefix. This is non-persistent but relevant to updating the UI. Example is updated in the showcase. It must be combined with the relatively new :refresh-suffixes prefix slot.

The real issue I think is the granularity of the infix persistence. Right now transient's persistence will save / set the entire transient? Having automatic set / save on an infix would thus clobber any other infix without :unsaveable t, which is also very inflexible for that infix.

The :init-value slot can provide a way to evade the normal save persistence. Overriding transient-init-value or even scope is another way to blend your custom persistence into the normal methods. I have some examples in the Transient Showcase. The custom infix class implements its own re-hydration, which reads history of the prefix.

As a package developer, sometimes it makes sense to just put transient-set as an option in the transient, just to remind the user to persist frequently changed values. That can help with infixes not being remembered. C-g cancels the settings change, and adding transient-save as an explicit user action are intuitive if not perfect.

By the looks of it, @karthink has made use of :description slots. If you're providing feedback that way, it might be faster just to serialize values to your own .el file and implement infixes as suffixes. At that point, transient is mostly a frontend and key binding system, but it's completely flexible to use it that way and doesn't require junking up transient.

A face value specific for values that are "sticky" would be helpful to show the user which settings must be set/saved versus which ones are automatically persisting to some degree. Even when we are doing completely custom persistence, this is important for UX.