Merge pull request #150 from KingSan666888/develop/v0.2.2

add CoT operator and example

examples/cot/README.md

@@ -0,0 +1,109 @@
+# Chain-of-Thought (CoT) Example
+
+Chain-of-Thought (CoT) is a reasoning approach that involves breaking down a complex problem into a series of intermediate steps or thoughts. This method allows for a more structured and transparent problem-solving process, where each step builds upon the previous one to reach a final solution. By explicitly outlining the thought process, CoT helps in understanding and solving intricate tasks more effectively.
+
+
+This example demonstrates how to use the framework for cot tasks. The example code can be found in the `examples/cot` directory.
+
+```bash
+   cd examples/cot
+```
+
+## Overview
+
+This example implements a cot workflow that consists of following components:
+
+1. **Input Interface**
+   - Input CoT Method： Choose between `few_shot` or `zero_shot` methods for the Chain-of-Thought process.
+   - Input Sample Examples: Provide sample examples if using the `few_shot` method to guide the reasoning process.
+   - Input Question: Specify the question or task that needs to be solved using the CoT approach.
+
+2. **CoT Workflow**
+   - Analyze and reason through to derive the final answer.
+
+### This whole workflow is looked like the following diagram:
+
+<div style="text-align: center;">
+    <img src="./docs/images/cot_workflow_diagram.png" alt="CoT Workflow" width="300"/>
+</div>
+
+## Prerequisites
+
+- Python 3.10+
+- Required packages installed (see requirements.txt)
+- Access to OpenAI API or compatible endpoint (see configs/llms/*.yml)
+- Redis server running locally or remotely
+- Conductor server running locally or remotely
+
+## Configuration
+
+The container.yaml file is a configuration file that manages dependencies and settings for different components of the system, including Conductor connections, Redis connections, and other service configurations. To set up your configuration:
+
+1. Generate the container.yaml file:
+   ```bash
+   python compile_container.py
+   ```
+   This will create a container.yaml file with default settings under `examples/cot`.
+
+
+2. Configure your LLM settings in `configs/llms/*.yml`:
+   - Set your mdoel_id or OpenAI API key or compatible endpoint through environment variable or by directly modifying the yml file
+   ```bash
+   export custom_model_id="your_model_id"
+   export custom_openai_key="your_openai_api_key"
+   export custom_openai_endpoint="your_openai_endpoint"
+   ```
+
+   - Configure other model settings like temperature as needed through environment variable or by directly modifying the yml file
+
+3. Update settings in the generated `container.yaml`:
+   - Modify Redis connection settings:
+     - Set the host, port and credentials for your Redis instance
+     - Configure both `redis_stream_client` and `redis_stm_client` sections
+   - Update the Conductor server URL under conductor_config section
+   - Adjust any other component settings as needed
+
+## Running the Example
+
+1. Run the CoT example:
+
+   For terminal/CLI usage:
+   ```bash
+   python run_cli.py
+   ```
+
+   For app/GUI usage:
+   ```bash
+   python run_app.py
+   ```
+
+   For webpage/GRADIO usage:
+   ```bash
+   python run_webpage.py
+   ```
+
+2. Run the evaluation example:
+   - you need to set the parameters such as `model_id`, `dataset_name`, `dataset_path`, `output_path`, `output_name`, and `cot_method`, which are defined as follows:
+      - `model_id`: The ID of the LLM model you are using.
+      - `dataset_name`: The name of the dataset.
+      - `dataset_path`: The path to the dataset.
+      - `output_path`: The path where the output will be saved.
+      - `output_name`: The name of the output file.
+      - `cot_method`: The Chain-of-Thought method to use (e.g., `few_shot` or `zero_shot`).
+
+   - ❗**NOTE**: You may need to modify the `prepare_data` function to suit your specific dataset.
+
+   After setting the parameters, run the following command to start the evaluation:
+   ```bash
+   python eval_demo.py --model_id your_model_id --dataset_name your_dataset_name --dataset_path your_dataset_path --output_path your_output_path --output_name your_output_name --cot_method your_cot_method
+
+## Troubleshooting
+
+If you encounter issues:
+- Verify Redis is running and accessible
+- Check your OpenAI API key is valid
+- Check your Bing API key is valid if search results are not as expected
+- Ensure all dependencies are installed correctly
+- Review logs for any error messages
+- **Open an issue on GitHub if you can't find a solution, we will do our best to help you out!**
+

examples/cot/agent/input_interface/input_interface.py

@@ -0,0 +1,69 @@
+from pathlib import Path
+
+from omagent_core.utils.registry import registry
+from omagent_core.utils.general import read_image
+from omagent_core.engine.worker.base import BaseWorker
+from omagent_core.utils.logger import logging
+
+CURRENT_PATH = Path( __file__ ).parents[ 0 ]
+
+
+@registry.register_worker()
+class InputInterface( BaseWorker ):
+    """Input interface processor that handles user instructions and image input.
+    
+    This processor:
+    1. Reads user input containing question and image via input interface
+    2. Extracts text instruction and image path from the input
+    3. Loads and caches the image in workflow storage
+    4. Returns the user instruction for next steps
+    """
+
+    def _run( self, *args, **kwargs ):
+        # Read user input through configured input interface
+        user_input = self.input.read_input(
+            workflow_instance_id=self.workflow_instance_id,
+            input_prompt='Please provide your method (few_shot or zero_shot):'
+        )
+        messages = user_input[ 'messages' ]
+        cot_method = messages[ -1 ][ 'content' ][ 0 ][ 'data' ]
+
+        assert cot_method in [ 'few_shot', 'zero_shot' ], "Invalid method provided"
+
+        if cot_method == 'few_shot':
+            user_input = self.input.read_input(
+                workflow_instance_id=self.workflow_instance_id,
+                input_prompt=
+                'If using few_shot method, please provide your examples (in the order of question, reasoning, answer). If using zero_shot, please press enter to skip:'
+            )
+            messages = user_input[ 'messages' ]
+            message = messages[ -1 ][ 'content' ]
+
+            assert len( message ) % 3 == 0, "Invalid number of examples provided"
+            cot_examples = [
+                {
+                    "q": example[ 0 ][ 'data' ],
+                    "r": example[ 1 ][ 'data' ],
+                    "a": example[ 2 ][ 'data' ]
+                } for example in
+                [ message[ i : i + 3 ] for i in range( 0, len( message ), 3 ) ]
+            ]
+        else:
+            cot_examples = []
+
+        user_input = self.input.read_input(
+            workflow_instance_id=self.workflow_instance_id,
+            input_prompt='Please provide your question:'
+        )
+        messages = user_input[ 'messages' ]
+        query = messages[ -1 ][ 'content' ][ 0 ][ 'data' ]
+
+        logging.info(
+            f"InputInterface: query={query}, cot_method={cot_method}, cot_examples={cot_examples}"
+        )
+
+        return {
+            'query': query,
+            'cot_method': cot_method,
+            'cot_examples': cot_examples
+        }

examples/cot/compile_container.py

@@ -0,0 +1,17 @@
+# Import core modules and components
+from omagent_core.utils.container import container
+
+# Import workflow related modules
+from pathlib import Path
+from omagent_core.utils.registry import registry
+
+# Set up path and import modules
+CURRENT_PATH = root_path = Path(__file__).parents[0]
+registry.import_module()
+
+# Register required components
+container.register_callback(callback='AppCallback')
+container.register_input(input='AppInput')
+container.register_stm("RedisSTM")
+# Compile container config
+container.compile_config(CURRENT_PATH)

examples/cot/configs/llms/gpt4o.yml

@@ -0,0 +1,8 @@
+name: OpenaiGPTLLM
+model_id: ${env| custom_model_id, model_id} # gpt-4o-mini
+api_key: ${env| custom_openai_key, openai_api_key}
+endpoint: ${env| custom_openai_endpoint, https://api.openai.com/v1}
+temperature: 0
+vision: true
+# response_format: json_object
+use_default_sys_prompt: False

examples/cot/configs/workers/cot_workflow.yml

@@ -0,0 +1,5 @@
+- name: CoTReasoning
+  llm: ${sub|gpt4o}
+  output_parser: 
+    name: StrParser
+  concurrency: 15

examples/cot/configs/workers/input_interface.yml

@@ -0,0 +1 @@
+name: InputInterface

examples/cot/eval_demo.py

@@ -0,0 +1,164 @@
+import os
+import json
+import argparse
+
+from pathlib import Path
+
+from omagent_core.utils.logger import logging
+from omagent_core.utils.registry import registry
+from omagent_core.utils.container import container
+from omagent_core.engine.workflow.conductor_workflow import ConductorWorkflow
+from omagent_core.advanced_components.workflow.cot.workflow import CoTWorkflow
+from omagent_core.clients.devices.programmatic.client import ProgrammaticClient
+
+EXAMPLES = [
+    {
+        "q":
+            "There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?",
+        "r":
+            "There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6.",
+        "a":
+            "6"
+    }, {
+        "q": "If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?",
+        "r": "There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5.",
+        "a": "5"
+    }, {
+        "q": "Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?",
+        "r": "Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 + 42 = 74. After eating 35, they had 74 - 35 = 39.",
+        "a": "39"
+    }, {
+        "q": "Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?",
+        "r": "Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12 = 8.",
+        "a": "8"
+    }, {
+        "q": "Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?",
+        "r": "Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then that is 4 more toys. 5 + 4 = 9.",
+        "a": "9"
+    }, {
+        "q":
+            "There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?",
+        "r":
+            "There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29.",
+        "a":
+            "29"
+    }, {
+        "q":
+            "Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday?",
+        "r":
+            "Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 - 23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls.",
+        "a":
+            "33"
+    }, {
+        "q": "Olivia has $23. She bought five bagels for $3 each. How much money does she have left?",
+        "r": "Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. So she has 23 - 15 dollars left. 23 - 15 is 8.",
+        "a": "8"
+    }
+]
+
+
+def read_jsonl( file_path ):
+    with open( file_path, 'r', encoding='utf-8' ) as file:
+        data = [ json.loads( line ) for line in file ]
+    return data
+
+
+def write_json( data, file_path, file_name ):
+    file = os.path.join( file_path, f"{file_name}.json" )
+    with open( file, 'w' ) as f:
+        json.dump( data, f, indent=4 )
+    return file
+
+
+def prepare_data( file_path, cot_examples, cot_method, start=1 ):
+    data = read_jsonl( file_path )
+
+    return [
+        {
+            "id": idx,
+            "query": d[ 'question' ],
+            "cot_method": cot_method,
+            "cot_examples": cot_examples
+        } for idx, d in enumerate( data, start=start )
+    ]
+
+
+def process_results( res, dataset_name, alg ):
+    return { "dataset": dataset_name, "model_id": os.environ.get( 'custom_model_id' ), "alg": alg, "model_result": res}
+
+
+def setup_environ( model_id ):
+    os.environ[ 'custom_model_id' ] = model_id
+
+    print( f"Model ID: {model_id}" )
+
+
+def evaluator( model_id, dataset_name, dataset_path, output_path, output_name, cot_method, alg='COT' ):
+    # Set up the evaluator
+    setup_environ( model_id=model_id )
+
+    # Initialize logging
+    logging.init_logger( "omagent", "omagent", level="INFO" )
+
+    # Set current working directory path
+    CURRENT_PATH = Path( __file__ ).parents[ 0 ]
+
+    # Import registered modules
+    registry.import_module( project_path=CURRENT_PATH.joinpath( 'agent' ) )
+
+    container.register_stm( "RedisSTM" )
+    # Load container configuration from YAML file
+    container.from_config( CURRENT_PATH.joinpath( 'container.yaml' ) )
+
+    workflow = ConductorWorkflow( name='cot_eval' )
+
+    cot_workflow = CoTWorkflow()
+    cot_workflow.set_input(
+        id=workflow.input( 'id' ),
+        query=workflow.input( 'query' ),
+        cot_method=workflow.input( 'cot_method' ),
+        cot_examples=workflow.input( 'cot_examples' )
+    )
+
+    workflow >> cot_workflow
+
+    config_path = CURRENT_PATH.joinpath( "configs" )
+    programmatic_client = ProgrammaticClient( processor=workflow, config_path=config_path, workers=[] )
+
+    workflow_input_list = prepare_data( file_path=dataset_path, cot_examples=EXAMPLES, cot_method=cot_method )
+    res = programmatic_client.start_batch_processor( workflow_input_list=workflow_input_list, max_tasks=15 )
+    programmatic_client.stop_processor()
+
+    results = process_results( res, dataset_name=dataset_name, alg=alg )
+    file = write_json( data=results, file_path=output_path, file_name=output_name )
+
+    print( f"{dataset_name} evaluation results saved in {file}" )
+
+
+def arguments():
+    parser = argparse.ArgumentParser( description='COT Evaluation' )
+
+    parser.add_argument( '--cot_method', type=str, default='zero_shot', help='COT Method' )
+    parser.add_argument( '--model_id', type=str, default='gpt-3.5-turbo', help='Model ID' )
+
+    parser.add_argument( '--dataset_name', type=str, default='gsm8k', help='Dataset Name' )
+    parser.add_argument( '--dataset_path', type=str, default='gsm8k.jsonl', help='Dataset Path' )
+
+    parser.add_argument( '--output_path', type=str, default='Output', help='Output Path' )
+    parser.add_argument( '--output_name', type=str, default='gsm8k_results', help='Output Name' )
+
+    print( f"Arguments: {parser.parse_args()}" )
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    args = arguments()
+
+    evaluator(
+        model_id=args.model_id,
+        dataset_name=args.dataset_name,
+        dataset_path=args.dataset_path,
+        output_path=args.output_path,
+        output_name=args.output_name,
+        cot_method=args.cot_method
+    )