Optional[...] = None in Pydantic model froze at compiling FSM in outlines.generate.json(model, schema)

[whatswrongwithyourmitochondria]

Describe the issue as clearly as possible:

I got a problem while creating a generator with outlines.generate.json(model, schema) and as a schema using Pydantic Model with the fields like name: Optional[...] = Field(default=None,...). It froze at compiling FSM, basically, at crawl() computing map for the state (from interegular lib).

In the example provided I put 1/4 of the Pydantic Model I have, overall schema is massive.
I was able to fix that part getting rid of default=None in Optional[...] fields.
After removing it, the generator was created in seconds.

Steps/code to reproduce the bug:

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, pipeline

from pydantic import BaseModel, Field, field_validator
from pydantic.json_schema import models_json_schema

import outlines
from outlines import models, generate, types
from typing import Literal, Optional, List
from enum import Enum

import pandas as pd
import json
import torch

from huggingface_hub import login
from google.colab import userdata
login(token=userdata.get('HF_TOKEN'))

torch.random.manual_seed(0)

model_name = "meta-llama/Llama-3.1-8B-Instruct"
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
                    model_name,
                    device_map="auto",
                    torch_dtype=torch.float16,
                    quantization_config=quantization_config
                    )

tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = models.Transformers(model, tokenizer)

class MorphologyType(str, Enum):
    solid_polypoid = "solid_polypoid"
    solid_semi_annular = "solid_semi_annular"
    mucinous = "mucinous"
    other = "OTHER"
    missing = "missing"

class PeritonealReflectionRelation(str, Enum):
    below = "below"
    above = "above"
    crossing = "crossing"
    missing = "missing"

class TStageType(str, Enum):
    t12 = "T1_2"
    t3ab = "T3ab"
    t3cd = "T3cd"
    t4 = "T4"
    missing = "missing"

class SphincterInvasionType(str, Enum):
    no = "no"
    internal_only = "yes, internal sphincter only"
    internal_intersphincteric= "yes, internal sphincter + intersphincteric plane"
    internal_external = "yes, internal + external sphincter"
    unknown = "unknown"

class InvasionPart(str, Enum):
    upper = "upper"
    middle = "middle"
    distal = "distal"
    other = "OTHER"
    na = "not applicable"


class LocalTumorStatus(BaseModel):
    morphology: MorphologyType = Field(
        default=MorphologyType.missing,
        alias='Morphology: [solid_polypoid/solid_semi_annular/mucinous]'
    )
    circumferential_tumor_involvement_start: Optional[int] = Field(
        default=None,
        alias='Circumferential tumor involvement from [ ] o’clock'
    )
    circumferential_tumor_involvement_end: Optional[int] = Field(
        default=None,
        alias='Circumferential tumor involvement: to [ ] o’clock'
    )
    distance_from_anorectal_junction_cm: Optional[float] = Field(
        default=None,
        alias='Distance from anorectal junction to lower pole of tumor in cm'
    )
    relation_to_anterior_peritoneal_reflection: PeritonealReflectionRelation = Field(
        default=PeritonealReflectionRelation.missing,
        alias='Relation to anterior peritoneal reflection: [below/above/crossing]'
    )
    tumor_length_cm: Optional[float] = Field(
        default=None,
        alias='Tumor length in cm'
    )
    t_stage: TStageType = Field(
        default=TStageType.missing,
        alias='T_stage: [T1_2/T3ab/T3cd/T4]'
    )

    growth_into: Optional[str] = Field(
        default=None,
        alias='If T4, based on growth into: [ ]'
    )

    sphincter_invasion: SphincterInvasionType = Field(
        default=SphincterInvasionType.unknown,
        alias='Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]'
    )
    lowest_part_of_sphincter_invasion: InvasionPart = Field(
        default=InvasionPart.na,
        alias='If yes, lowest part of invasion [upper/middle/distal]'
    )

report_processor = generate.json(llm, LocalTumorStatus)

Expected result:

getting a generator

Error message:

the execution 'route':
Executing (3m 8s) <cell line: 1> ➔ wrapper() ➔ json() ➔ wrapper() ➔ regex() ➔ __init__() ➔ from_regex() ➔ from_regex() ➔ wrapper() ➔ cached_create_states_mapping() ➔ create_states_mapping() ➔ to_fsm() ➔ <genexpr> ➔ to_fsm() ➔ to_fsm() ➔ __mul__() ➔ times() ➔ crawl()

Outlines/Python version information:

Outlines version
0.1.3
Python version
Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]

### Context for the issue:

The code is executed in Colab on NVIDIA L4 GPU. 

[whatswrongwithyourmitochondria]

Unfortunately, without a default value equaling None the model hallucinates and output random numbers in the corresponding fields.

[whatswrongwithyourmitochondria]

I checked the regex obtained from schema without the defaults and with the defaults. For some reason with the defaults regex gets much bigger and with repetitions:

regex without defaults:

Click to expand/collapse

\{([ ]?"Morphology:\ \[solid_polypoid/solid_semi_annular/mucinous\]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?[ ]?"Circumferential\ tumor\ involvement\ from\ \[\ \]\ o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,[ ]?"Circumferential\ tumor\ involvement:\ to\ \[\ \]\ o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,[ ]?"Distance\ from\ anorectal\ junction\ to\ lower\ pole\ of\ tumor\ in\ cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,([ ]?"Relation\ to\ anterior\ peritoneal\ reflection:\ \[below/above/crossing\]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?[ ]?"Tumor\ length\ in\ cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,([ ]?"T_stage:\ \[T1_2/T3ab/T3cd/T4\]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing")[ ]?,)?[ ]?"If\ T4,\ based\ on\ growth\ into:\ \[\ \]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null)([ ]?,[ ]?"Sphincter\ invasion:\ \[no/yes,\ internal\ sphincter\ only/yes,\ internal\ sphincter\ \+\ intersphincteric\ plane/yes,\ internal\ \+\ external\ sphincter\]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If\ yes,\ lowest\ part\ of\ invasion\ \[upper/middle/distal\]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?[ ]?\}

regex with defaults:

Click to expand/collapse

\{([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")([ ]?,[ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null))?([ ]?,[ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null))?([ ]?,[ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing"))?([ ]?,[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?[ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)([ ]?,[ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null))?([ ]?,[ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing"))?([ ]?,[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?[ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)([ ]?,[ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing"))?([ ]?,[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?[ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)([ ]?,[ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing"))?([ ]?,[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?[ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")([ ]?,[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null))?([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?[ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)([ ]?,[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing"))?([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?([ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?[ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing")([ ]?,[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null))?([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?([ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing")[ ]?,)?[ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null)([ ]?,[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown"))?([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?([ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing")[ ]?,)?([ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null)[ ]?,)?[ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown")([ ]?,[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?|([ ]?"Morphology: [solid_polypoid/solid_semi_annular/mucinous]"[ ]?:[ ]?("solid_polypoid"|"solid_semi_annular"|"mucinous"|"OTHER"|"missing")[ ]?,)?([ ]?"Circumferential tumor involvement from [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Circumferential tumor involvement: to [ ] o’clock"[ ]?:[ ]?((-)?(0|[1-9][0-9]*)|null)[ ]?,)?([ ]?"Distance from anorectal junction to lower pole of tumor in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"Relation to anterior peritoneal reflection: [below/above/crossing]"[ ]?:[ ]?("below"|"above"|"crossing"|"missing")[ ]?,)?([ ]?"Tumor length in cm"[ ]?:[ ]?(((-)?(0|[1-9][0-9]*))(\.[0-9]+)?([eE][+-][0-9]+)?|null)[ ]?,)?([ ]?"T_stage: [T1_2/T3ab/T3cd/T4]"[ ]?:[ ]?("T1_2"|"T3ab"|"T3cd"|"T4"|"missing")[ ]?,)?([ ]?"If T4, based on growth into: [ ]"[ ]?:[ ]?("([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"|null)[ ]?,)?([ ]?"Sphincter invasion: [no/yes, internal sphincter only/yes, internal sphincter + intersphincteric plane/yes, internal + external sphincter]"[ ]?:[ ]?("no"|"yes,\ internal\ sphincter\ only"|"yes,\ internal\ sphincter\ \+\ intersphincteric\ plane"|"yes,\ internal\ \+\ external\ sphincter"|"unknown")[ ]?,)?[ ]?"If yes, lowest part of invasion [upper/middle/distal]"[ ]?:[ ]?("upper"|"middle"|"distal"|"OTHER"|"not\ applicable"))?[ ]?\}