when choosing Florence-2-base-PromptGen-v1.5 model, error as bellow

[jasoncow007]

Traceback (most recent call last):
File "D:\ComfyUI-aki-v1.2\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "D:\ComfyUI-aki-v1.2\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "D:\ComfyUI-aki-v1.2\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "D:\ComfyUI-aki-v1.2\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "D:\ComfyUI-aki-v1.2\custom_nodes\ComfyUI-Florence2\nodes.py", line 97, in loadmodel
model = AutoModelForCausalLM.from_pretrained(model_path, attn_implementation=attention, device_map=device, torch_dtype=dtype,trust_remote_code=True)
File "D:\ComfyUI-aki-v1.2\python\lib\site-packages\transformers\models\auto\auto_factory.py", line 526, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "D:\ComfyUI-aki-v1.2\python\lib\site-packages\transformers\models\auto\configuration_auto.py", line 1025, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type florence2 but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

[Jeff-goal]

same problem, even after reinstalled the plugin

[askmyteapot]

So its an issue with the promptgen model's config file.

I got it to work by doing the following.

1. Edit config.json and add in the following at line 6

  "AutoConfig": "configuration_florence2.Florence2Config",
  "AutoModelForCausalLM": "modeling_florence2.Florence2ForConditionalGeneration"
  },

2. Copy from the subfolder florence2_large the 3 *_florence2.py files into the main folder for promtgen

It will re-download a newer file from the microsoft repo, then it will work.

[freshwind-bit]

@askmyteapot A small clarification for your instruction "Edit config.json and add the following at line 6":
not add at line 6, but insert before line 6. For a large 1.5 model it should look like this:

{
"_name_or_path": "microsoft/Florence-2-large",
"architectures": [
"Florence2ForConditionalGeneration"
],
"auto_map": {
"AutoConfig": "configuration_florence2.Florence2Config",
"AutoModelForCausalLM": "modeling_florence2.Florence2ForConditionalGeneration"
},
"bos_token_id": 0,
"eos_token_id": 2,

Accordingly, for the base 1.5 model it should look like this:

{
"_name_or_path": "microsoft/Florence-2-base-ft",
"architectures": [
"Florence2ForConditionalGeneration"
],
"auto_map": {
"AutoConfig": "configuration_florence2.Florence2Config",
"AutoModelForCausalLM": "modeling_florence2.Florence2ForConditionalGeneration"
},
"bos_token_id": 0
"eos_token_id": 2,

just like it is done in models\LLM\Florence-2-large\config.json

The confusion is added by the fact that in other extension packs (for example in LayerStyle) these models are downloaded to the florence2 directory, not to LLM, and these config.jsons do not contain the discussed code. Therefore, being in the models\florence2\ subdirectory, I simply create symlinks to the LLM subdirectories in PoweShell:

New-Item -ItemType SymbolicLink -Path ".\large" -Target "..\LLM\Florence-2-large"
New-Item -ItemType SymbolicLink -Path ".\base" -Target "..\LLM\Florence-2-base"
New-Item -ItemType SymbolicLink -Path ".\base-PromptGen" -Target "..\LLM\Florence-2-base-PromptGen"
New-Item -ItemType SymbolicLink -Path ".\base-PromptGen-v1.5" -Target "..\LLM\Florence-2-base-PromptGen-v1.5"
New-Item -ItemType SymbolicLink -Path ".\large-PromptGen-v1.5" -Target "..\LLM\Florence-2-large-PromptGen-v1.5"