Guide: ROSA Prompt Engineering

Tips & Tricks for Maximizing System Prompts Effectiveness

1. Prioritize Safety in Critical Instructions

   • Always focus on instructions that ensure safety first, both for the robot and its environment.
   • Use explicit fail-safes: e.g., "If unsure, halt and ask for operator input."

2. Reflect Real-World Context in Environment Descriptions

   • Tailor the environment description to dynamic elements that the robot will encounter regularly (e.g., moving objects, shifting lighting).
   • Keep environment updates frequent when operating in variable conditions.

3. Align Persona with Task Objectives

   • Make the robot's persona directly support its core functions. For instance, if a robot is assisting in customer service, embody a "helpful and courteous" persona.
   • Adjust the persona to match the operator's expectations, especially if they work in high-stress or fast-paced settings.

4. Use Scenarios to Clarify Mission and Objectives

   • Frame objectives with real-world scenarios that the robot will likely encounter (e.g., "Your objective is to clear debris from a narrow hallway when a blockage is detected").
   • Define multi-step goals in simple terms, ensuring the robot can adapt to interruptions without deviating from the mission.

5. Provide Layered Constraints for More Robust Guardrails

   • Combine both hard and soft constraints. For example, "You must never exceed a speed of 2 m/s in the presence of humans" (hard), and "Try to avoid working too close to fragile objects" (soft).
   • Include fallback behaviors when a constraint is hit (e.g., stop, alert operator, reattempt).

6. Be Descriptive, Not Prescriptive, About Capabilities

   • Rather than saying “You can perform X,” describe the context in which a capability is useful: "You can use your gripper to pick up objects with uneven surfaces such as boxes or bags."
   • Avoid over-defining actions, so the LLM has room to make informed decisions based on the situation.

7. Incorporate Operator Behavior in "About Your Operators"

   • Consider how operators might change behaviors under stress and include prompts to account for miscommunication or rushed commands (e.g., “If your operator issues a conflicting command, ask for clarification”).

8. Use Environmental Variables Dynamically

   • Adjust environmental variables based on specific operational phases or location-specific challenges (e.g., different ROS settings for indoors vs. outdoors).
   • Include fallback IPs or modes in case environment variables fail to load.

9. Make Nuances and Assumptions Clear for Edge Cases

   • Define subtle behaviors for edge cases (e.g., "If an object is partially in your path, you can nudge it aside if it weighs less than 1 kg").
   • Explicitly mention common failure modes and how the robot should interpret them.

10. Frequent Updates Based on Real-World Feedback

    • Iterate on prompts frequently by gathering data on how well the robot follows instructions in practice.
    • Use logs of failed actions to identify missing prompts or under-defined instructions that need refining.

---

Bonus: Testing Prompts in Simulation

• Test in a simulator first: Before deploying in real-world scenarios, simulate the system prompts to see how effectively the robot behaves.
• A/B Test persona tweaks: Experiment with slight adjustments to the robot's persona to see which variations produce better task outcomes.

---

These strategies will help ensure your system prompts are highly practical, adaptive, and relevant, ultimately leading to a more robust and capable robot-agent.