Skip to content

Commit

Permalink
added user stories from tutorial, changed way partially fulfilled gui…
Browse files Browse the repository at this point in the history
…delines are handled upon re-application
  • Loading branch information
MCBarKar committed Dec 10, 2024
1 parent 71c6b22 commit e2b57d1
Show file tree
Hide file tree
Showing 6 changed files with 102 additions and 20 deletions.
15 changes: 14 additions & 1 deletion src/parlant/core/engines/alpha/guideline_proposer.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ def _format_prompt(
Conversely, actions dictating one-time behavior (e.g., "send the user our address") should be re-applied more conservatively.
Only re-apply these if the condition ceased to be true earlier in the conversation before being fulfilled again in the current context.
IMPORTANT: Some guidelines include multiple actions. If only a portion of those actions were fulfilled earlier in the conversation, treat the guideline as though it has been fully executed. In such cases, re-apply the guideline only if its condition becomes true again later in the conversation, unless it is marked as continuous.
Examples of Condition Evaluations:
-------------------
Expand Down Expand Up @@ -443,7 +444,7 @@ def _format_prompt(
- Guidelines: ###
1) condition: the customer asks about the value of a stock. action: provide the price using the 'check_stock_price' tool
2) condition: the weather at a certain location is discussed. action: check the weather at that location using the 'check_weather' tool
3) condition: the customer asked about the weather. action: provide the customre with the temperature and the chances of precipitation
###
- **Expected Result**:
```json
Expand All @@ -467,6 +468,18 @@ def _format_prompt(
"condition_application_rationale": "while weather was discussed earlier, the conversation have moved on to an entirely different topic (stock prices)",
"applies_score": 3
}},
{{
"guideline_number": 3,
"condition": "the customer asked about the weather.",
"condition_applies": true,
"condition_application_rationale": "The customer asked about the weather earlier, though the conversation has somewhat moved on to a new topic",
"action": "provide the customre with the temperature and the chances of precipitation",
"guideline_is_continuous": false,
"guideline_previously_applied_rationale": "The action was partially fulfilled by reporting the temperature without the chances of precipitation. As partially fulfilled guidelines are treated as completed, this guideline is considered applied",
"guideline_previously_applied": true,
"guideline_should_reapply": false,
"applies_score": 4
}},
]
}}
```
Expand Down
2 changes: 1 addition & 1 deletion src/parlant/core/engines/alpha/message_event_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,7 @@ def _get_output_format(
"""
else:
insights_output_format = """
<Additional entires for all insights>
<Additional entries for all insights>
"""

return f"""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,38 @@ Feature: Conversation
Then a single message event is emitted
And the message contains 'productivity', 'professional', and 'development'

Scenario: The agent correctly applies greeting guidelines based on auxillary data
Given an agent named "Chip Bitman" whose job is to work at a tech store and help customers choose what to buy. You're clever, witty, and slightly sarcastic. At the same time you're kind and funny.
And a customer with the name "Beef Wellington"
And an empty session with "Beef Wellingotn"
And the term "Bug" defined as The name of our tech retail store, specializing in gadgets, computers, and tech services.
And the term "Bug-Free" defined as Our free warranty and service package that comes with every purchase and covers repairs, replacements, and tech support beyond the standard manufacturer warranty.
And a customer tagged as "tag::business"
And a context variable "plan" set to "Business Plan" for the tag "tag::business"
And a guideline to just welcome them to the store and ask how you can help when the customer greets you
And a guideline to refer to them by their first name only, and welcome them 'back' when a customer greets you
And a guideline to assure them you will escalate it internally and get back to them when a business-plan customer is having an issue
And a customer message, "Hi there"
When processing is triggered
Then a single message event is emitted
And the message contains the name 'Beef'
And the message contains a welcoming back of the customer to the store and asking how the agent could help

Scenario: The agent doesnt wrongly reapply partially fulfilled guideline
Given an agent named "Chip Bitman" whose job is to work at a tech store and help customers choose what to buy. You're clever, witty, and slightly sarcastic. At the same time you're kind and funny.
And a customer with the name "Beef Wellington"
And an empty session with "Beef Wellingotn"
And the term "Bug" defined as The name of our tech retail store, specializing in gadgets, computers, and tech services.
And the term "Bug-Free" defined as Our free warranty and service package that comes with every purchase and covers repairs, replacements, and tech support beyond the standard manufacturer warranty.
And a customer tagged as "tag::business"
And a context variable "plan" set to "Business Plan" for the tag "tag::business"
And a guideline to just welcome them to the store and ask how you can help when the customer greets you
And a guideline to refer to them by their first name only, and welcome them 'back' when a customer greets you
And a guideline to assure them you will escalate it internally and get back to them when a business-plan customer is having an issue
And a customer message, "Hi there"
And an agent message, "Hey Beef, welcome to Bug! How can I help you today?"
And a customer message, "I'm having issues with my web camera"
When processing is triggered
Then a single message event is emitted
And the message contains no welcoming back of the customer
And the message contains that the request will be escelated
19 changes: 19 additions & 0 deletions tests/core/engines/alpha/steps/agents.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,22 @@ def given_an_agent_with_identity(
)
)
return agent.id


@step(
given,
parsers.parse('an agent named "{name}" whose job is {description}'),
target_fixture="agent_id",
)
def given_an_agent_with_identity_and_name(
context: ContextOfTest,
description: str,
name: str,
) -> AgentId:
agent = context.sync_await(
context.container[AgentStore].create_agent(
name=name,
description=f"Your job is {description}",
)
)
return agent.id
50 changes: 32 additions & 18 deletions tests/core/engines/alpha/steps/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
from typing import Optional, cast
from pytest_bdd import given, then, parsers, when

from parlant.core.agents import Agent
from parlant.core.agents import AgentId, AgentStore
from parlant.core.common import JSONSerializable
from parlant.core.customers import Customer
from parlant.core.customers import CustomerStore
from parlant.core.engines.alpha.utils import emitted_tool_event_to_dict
from parlant.core.emissions import EmittedEvent
from parlant.core.nlp.moderation import ModerationTag
Expand Down Expand Up @@ -45,10 +45,13 @@ def given_an_agent_message(
context: ContextOfTest,
agent_message: str,
session_id: SessionId,
agent: Agent,
agent_id: AgentId,
) -> SessionId:
store = context.container[SessionStore]
session = context.sync_await(store.read_session(session_id=session_id))
session_store = context.container[SessionStore]
agent_store = context.container[AgentStore]

session = context.sync_await(session_store.read_session(session_id=session_id))
agent = context.sync_await(agent_store.read_agent(agent_id))

message_data: MessageEventData = {
"message": agent_message,
Expand All @@ -59,7 +62,7 @@ def given_an_agent_message(
}

event = context.sync_await(
store.create_event(
session_store.create_event(
session_id=session.id,
source="ai_agent",
kind="message",
Expand All @@ -82,10 +85,13 @@ def given_a_human_message_on_behalf_of_the_agent(
context: ContextOfTest,
agent_message: str,
session_id: SessionId,
agent: Agent,
agent_id: AgentId,
) -> SessionId:
store = context.container[SessionStore]
session = context.sync_await(store.read_session(session_id=session_id))
session_store = context.container[SessionStore]
agent_store = context.container[AgentStore]

session = context.sync_await(session_store.read_session(session_id=session_id))
agent = context.sync_await(agent_store.read_agent(agent_id))

message_data: MessageEventData = {
"message": agent_message,
Expand All @@ -96,7 +102,7 @@ def given_a_human_message_on_behalf_of_the_agent(
}

event = context.sync_await(
store.create_event(
session_store.create_event(
session_id=session.id,
source="human_agent_on_behalf_of_ai_agent",
kind="message",
Expand All @@ -115,10 +121,12 @@ def given_a_customer_message(
context: ContextOfTest,
session_id: SessionId,
customer_message: str,
customer: Customer,
) -> SessionId:
store = context.container[SessionStore]
session = context.sync_await(store.read_session(session_id=session_id))
session_store = context.container[SessionStore]
customer_store = context.container[CustomerStore]

session = context.sync_await(session_store.read_session(session_id=session_id))
customer = context.sync_await(customer_store.read_customer(customer_id=session.customer_id))

message_data: MessageEventData = {
"message": customer_message,
Expand All @@ -129,7 +137,7 @@ def given_a_customer_message(
}

event = context.sync_await(
store.create_event(
session_store.create_event(
session_id=session.id,
source="customer",
kind="message",
Expand All @@ -154,18 +162,24 @@ def given_a_flagged_customer_message(
customer_message: str,
moderation_tag: ModerationTag,
) -> SessionId:
store = context.container[SessionStore]
session = context.sync_await(store.read_session(session_id=session_id))
session_store = context.container[SessionStore]
customer_store = context.container[CustomerStore]

session = context.sync_await(session_store.read_session(session_id=session_id))
customer = context.sync_await(customer_store.read_customer(customer_id=session.customer_id))

message_data: MessageEventData = {
"message": customer_message,
"participant": {"display_name": ""},
"participant": {
"id": customer.id,
"display_name": customer.name,
},
"flagged": True,
"tags": [moderation_tag],
}

event = context.sync_await(
store.create_event(
session_store.create_event(
session_id=session.id,
source="customer",
kind="message",
Expand Down
1 change: 1 addition & 0 deletions tests/core/engines/alpha/test_user_story_scenarios.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"sessions",
"terms",
"tools",
"customers",
)

scenarios(
Expand Down

0 comments on commit e2b57d1

Please sign in to comment.