23.md

Results of Code LLama 34B on HumanEval/23

Date: 01 September, 2023
Model: codellama-34b-instruct
Temperature: 0.2
Passing runs: 10
Total runs: 10
Success rate: 100%

Prompt

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """

The above is an incomplete Python code fragment. Return the complete and correct code with no additional text.

Correct solution

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """return len(string)

Unit tests

METADATA = {
    'author': 'jt',
    'dataset': 'test'
}


def check(candidate):
    assert candidate('') == 0
    assert candidate('x') == 1
    assert candidate('asdasnakj') == 9

Runs

✅ Run 1

Start time: 2023-09-01 02:48:17
Duration: 4.32s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 2

Start time: 2023-09-01 02:51:18
Duration: 4.87s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 3

Start time: 2023-09-01 02:54:13
Duration: 5.01s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 4

Start time: 2023-09-01 02:57:11
Duration: 3.77s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 5

Start time: 2023-09-01 03:00:10
Duration: 4.17s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 6

Start time: 2023-09-01 03:03:12
Duration: 6.36s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 7

Start time: 2023-09-01 03:06:14
Duration: 4.37s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 8

Start time: 2023-09-01 03:09:20
Duration: 4.76s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 9

Start time: 2023-09-01 03:12:12
Duration: 4.35s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)

✅ Run 10

Start time: 2023-09-01 03:15:17
Duration: 3.92s
Result: Pass

def strlen(string: str) -> int:
    """ Return length of given string
    >>> strlen('')
    0
    >>> strlen('abc')
    3
    """
    return len(string)