Assistant response post-processing

[cpursley]

Ability to automatically run additional post-processing on the last :assistant response.

Examples:

• Run a data extraction on the response (my use case is generating list of followup questions)
• Pipe contents of last message following a successful tool results call into a better/different chat model
• Run a quality/fact/formatting check on the contents of the last assistant response

[cpursley]

💻 It appears there's some related work on this branch: https://github.com/brainlid/langchain/tree/me-destructuring

So maybe this could be expanded to optionally take in different llm models, as some are better suited for certain tasks than others.

[cpursley]

The tool call is currently making an http call to 3rd party services. But I'be got plans for tools that ecto queries as well (like the fitness app). Main thing I'm after is the tool calling model I'm using isn't great at general purpose llm things and I want to process the tool call results with the better llm.

[cpursley]

Speaking of routing, it would also be awesome (if feasible) to have a smart prompt router that worked per prompt. For example, do a quick analysis on the user prompt and then decide which different model to use (if any). But all on the same chain instead of setting up a new chain. Not sure if that makes sense. Mainly, some models are better at certain tasks and being able to dynamically switch between them would be useful.

[brainlid]

@cpursley in my experience, a smaller/faster/lesser model doesn't do as well with tool calling. I think it will work better to have a single simple router chain examine the user's prompt and decide where it should go from there. Just route it to a chain setup for the purpose.

The selected chain can have the higher quality model you want and the appropriate tools loaded for it's purpose. You re-run the user's original prompt in the appropriate chain and let it do the ToolCalls instead of having that done on a less capable model.

[cpursley]

Funny, I came to a similar conclusion but just without the router. Just run one chain first that can do function calling and feed those results (I used :until_success and mapped out the ToolResult content into the stronger llm's prompt). Works pretty well.

The struggle I'm having is llama tool calling model I'm using kindof sucks for everything else (it's a one-trick pony). But giving those tool call json results to regular llama instruct model works well.

[brainlid]

@cpursley

it would also be awesome (if feasible) to have a smart prompt router that worked per prompt. For example, do a quick analysis on the user prompt and then decide which different model to use (if any). But all on the same chain instead of setting up a new chain.

I think you want an Agent.

Many smaller self-hosted models do 1 thing well 1 time. Meaning I can ask it to look at the user's prompt and have it decide if the user wants setup_1, setup_2 or setup_3. If you try to ask a follow-up question on that same model/chain, it will hallucinate to no end. It can do the 1 job 1 time. Any future uses need to start over.

I think what you really want is an Agent. The Agent internally manages multiple chains for multiple purposes but you only talk to the one Agent.

[brainlid]

Yes, with v0.3.0-rc.0 (what's currently on main), a new feature called MessageProcessors was added. A JsonProcessor is provided but custom ones can be created. The purpose is to process an Assistant response message to parse, change formatting, enforce rules, etc. When the assistant's response doesn't meet a rule, the processor generates a user message reporting the error to let the LLM try again.