OutSystems Developer Cloud (ODC) Custom Code Analyzer: Generator

ℹ️ This component is unofficial and unsupported by OutSystems.

⚡ CustomCode-Analyzer-Generator is a command line tool to generate ODC External Library solutions complete with unit tests. See examples of generated solutions here (> Connect to a redis database set up on my secure gateway (port 6379). password and key will be input params. return the key value) and here (> Take a python program in a string as input and return the stdout in the output). ⚡

Overview

The BSD-3 licensed CustomCode-Analyzer component was released to give you real-time feedback your C# code as you build an external library in your IDE of choice. But more generally CustomCode-Analyzer can be thought of as a way of locally validating an external library at the point of build. This makes it a powerful component in an end-to-end Large Language Model (LLM) pipeline for generating an ODC external library:

The CustomCode-Analyzer-Generator is an orchestrator around the CustomCode-Analyzer and other local components and external APIs. As seen in the above diagram the CustomCode-Analyzer-Generator takes user input for an external library functionality and:

• Calls the NuGet API and LLMs to generate the candidate implementation and unit test code.
• Creates implementation and unit test C# projects, installs packages and incorporates candidate code.
• Uses CustomCode-Analyzer as part of build to validate implementation code.
• Uses generated unit tests to further validate implementation code.
• Manages a reflection mechanism for self-correction and improvement in case of build failure or unit test failure.

In practice, that looks something like this (no speedup applied on a slow laptop!):

How to use

ℹ️ For safety, CustomCode-Analyzer-Generator runs inside a sandboxed Docker container. Therefore, you must first install Rancher Desktop (recommended) or Docker Desktop to run it.

Windows

1. Open Windows PowerShell.

2. Run the quick start script:

    irm https://raw.githubusercontent.com/jonathanalgar/CustomCode-Analyzer-Generator/refs/heads/main/windows_setup.ps1 | iex
   

3. Follow the instructions in the script.

Mac

(Quick start script for Mac coming soon).

1. Open the Terminal app.

2. Create the required directories in the home directory or elsewhere:

    mkdir -p ~/CustomCode-Analyzer-Generator/generated-solutions && chmod 700 ~/CustomCode-Analyzer-Generator/generated-solutions && cd ~/CustomCode-Analyzer-Generator
   

3. Create a .env file in ~/CustomCode-Analyzer-Generator and edit with your text editor of choice. Use .env_example as template.

4. Run the command:

    docker run --pull always -it --rm --env-file .env -v "$(pwd)/generated_solutions:/app/output" ghcr.io/jonathanalgar/customcode-analyzer-generator:latest
   

5. The generated solution libraries will be created in ~/CustomCode-Analyzer-Generator/generated_solutions.

Technical notes on evaluation

To validate that the LLM pipeline works effectively and generates external libraries aligned with user intent there is a supplementary evaluation pipeline complete with eight diverse test cases. The objective is to add more.

This means a range of models and prompts can be programatically benchmarked. The central piece of pipeline is the Roslyn-based CCGATestGenerator, which—given the LLM-generated implementation code and human created ground truth file—will deterministically create a unit test.

Test cases covered include binary data and network access through the secure gateway. The test set is currently limited to single action (but with support for multi-parameter using an LLM to do name mapping) and single type output.

Consolidated no-repetition results for 2x OpenAI models with one-shot prompt (current default prompt):

See full runs here or use this notebook.

You can replicate a run using:

docker compose -f run_full_benchmark.yml up --abort-on-container-exit

💡 API access to o3-mini is still in preview.

(More documentation and analysis coming soon.)

TODO

See here.

Feedback and contributions

Please report bugs and feature requests here.

PRs are welcome. In particular, code quality improvements, new unit/integration tests, new ground truths, and documentation improvements are all welcome 🤗 All changes to code should pass all existing tests (which are lacking and need to be expanded!). Please format any new code with Mypy and Flake8 for Python and Csharpier for C#.