Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System Prompt Does Not Utilize dspy.Signature Docstring (__doc__) #1836

Closed
yanmxa opened this issue Nov 21, 2024 · 5 comments · Fixed by #1840
Closed

System Prompt Does Not Utilize dspy.Signature Docstring (__doc__) #1836

yanmxa opened this issue Nov 21, 2024 · 5 comments · Fixed by #1840

Comments

@yanmxa
Copy link
Contributor

yanmxa commented Nov 21, 2024 •
edited
Loading

The dspy.Signature class docstring (doc) is not dynamically parsed or integrated into the system prompt.

Steps to Reproduce:

  1. Define a dspy.Signature subclass with a detailed docstring, e.g.,:
class KubeEngineer(dspy.Signature):
    """
    You are a Kubernetes engineer. Your task is to solve the given issue or perform the actions to interact with the Kubernetes cluster. 
    Use tools or kubectl commands to achieve the desired outcomes.
    
    Try to identify the following variables from the provided task or issue:
    - Cluster name
    - Resource type (e.g., Pod, Service, Deployment)
    - Resource name
    - Namespace
    """

    task: str = dspy.InputField(
        desc="The task or issue to address within the Kubernetes cluster."
    )
    answer: str = dspy.OutputField(
        desc="The final solution or output for the given task."
    )
  1. Run the following code:
lm = dspy.LM(
    model="llama-3.1-70b-versatile",
    api_base="https://api.groq.com/openai/v1",
    api_key=os.getenv("GROQ_API_KEY"),
)
with dspy.context(lm=lm):
    react = dspy.ReAct(KubeEngineer, tools=[kube_executor_with_configs.kubectl_cmd])
    result = react(
        task="***"
    )
print(lm.history)
  1. The system prompt in the output
{
  "prompt": None,
  "messages": [
    {
      "role": "system",
      "content": "Your input fields are:\n1. `task` (str): The task or issue to address within the Kubernetes cluster.\n2. `trajectory` (str)\n\nYour output fields are:\n1. `reasoning` (str)\n2. `answer` (str): The final solution or output for the given task.\n\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## task ## ]]\n{task}\n\n[[ ## trajectory ## ]]\n{trajectory}\n\n[[ ## reasoning ## ]]\n{reasoning}\n\n[[ ## answer ## ]]\n{answer}\n\n[[ ## completed ## ]]\n\nIn adhering to this structure, your objective is: \n        Given the fields `task`, produce the fields `answer`."
    },
    {
      "role": "user",
      "content": "...."
    }
  ]
}

Current Behavior:

The system prompt does not include the docstring from the KubeEngineer class.

Expected Output:

image 
{
  "prompt": None,
  "messages": [
    {
      "role": "system",
      "content": "You are a Kubernetes engineer. Your task is to solve the given issue or perform the actions to interact with the Kubernetes cluster. Use tools or kubectl commands to achieve the desired outcomes.\n\nTry to identify the following variables from the provided task or issue:\n- Cluster name\n- Resource type (e.g., Pod, Service, Deployment)\n- Resource name\n- Namespace ..."
    },
    {"role": "user", "content": "...."}
  ]
}
@yanmxa yanmxa mentioned this issue Nov 21, 2024
Closed
@okhat
Copy link
Collaborator

okhat commented Nov 21, 2024

Hey @yanmxa ! Thank you so much for opening an issue & PR for this.

I think you're observing this in ReAct only, not in general. The issue can be addressed by editing predict/react.py

@okhat
Copy link
Collaborator

okhat commented Nov 21, 2024

We need to pass instructions to this constructor:

dspy/dspy/predict/react.py

Line 67 in 698d0ac

dspy.Signature({**signature.input_fields, **signature.output_fields})

@yanmxa
Copy link
Contributor Author

yanmxa commented Nov 21, 2024

Thanks, @okhat! You got it! This issue occurs when using the ReAct module. Are there plans to address this? If so, I can draft a PR to resolve it.

@okhat
Copy link
Collaborator

okhat commented Nov 22, 2024

Thank you, @yanmxa ! A PR would be welcome :D The change should be a one-liner but you may need to check that it does work.

For line 67, it should become:

dspy.Signature({**signature.input_fields, **signature.output_fields}, signature.instructions)

@yanmxa yanmxa mentioned this issue Nov 22, 2024
Merged
@yanmxa
Copy link
Contributor Author

yanmxa commented Nov 22, 2024

@okhat Thank you for the prompt response!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@okhat @yanmxa