modular_model_converter cannot handle local imports with return

[xiuqhou]

System Info

• transformers version: 4.49.0.dev0
• Platform: Linux-5.15.0-70-generic-x86_64-with-glibc2.31
• Python version: 3.11.9
• Huggingface_hub version: 0.28.1
• Safetensors version: 0.4.5
• Accelerate version: 0.34.2
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (GPU?): 2.1.1 (True)
• Tensorflow version (GPU?): 2.15.1 (True)
• Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
• Jax version: 0.4.13
• JaxLib version: 0.4.13
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: No
• GPU type: NVIDIA GeForce RTX 3090

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Create a new folder named xxx_model in src/transformers/models/
2. Inside this folder, create a new Python file called modular_xxx.py with the following content:

from transformers.models.detr.image_processing_detr import DetrImageProcessor


class TmpImageProcessor(DetrImageProcessor):
    pass

3. Run the following command to execute the model converter:

python utils/modular_model_converter.py --files_to_parse src/transformers/models/xxx_model/modular_xxx.py

Expected behavior

The expected behavior is that it creates a file src/transformers/models/xxx_model/image_processing_xxx.py. However, the script fails with the following traceback:

Traceback (most recent call last):
  File "/Users/houxiuquan/Downloads/transformers/utils/modular_model_converter.py", line 1726, in <module>
    converted_files = convert_modular_file(file_name)
  File "/Users/houxiuquan/Downloads/transformers/utils/modular_model_converter.py", line 1663, in convert_modular_file
    for file, module in create_modules(cst_transformers).items():
  File "/Users/houxiuquan/Downloads/transformers/utils/modular_model_converter.py", line 1643, in create_modules
    needed_imports = get_needed_imports(body, all_imports)
  File "/Users/houxiuquan/Downloads/transformers/utils/modular_model_converter.py", line 1151, in get_needed_imports
    append_new_import_node(stmt_node, unused_imports, added_names, new_statements)
  File "/Users/houxiuquan/Downloads/transformers/utils/modular_model_converter.py", line 1111, in append_new_import_node
    for name in import_node.names:
AttributeError: 'Return' object has no attribute 'names'

I found the error is caused by the local imports with return in the following function of transformer.models.detr.image_processing_detr:

def get_numpy_to_framework_fn(arr) -> Callable:
    """
    Returns a function that converts a numpy array to the framework of the input array.

    Args:
        arr (`np.ndarray`): The array to convert.
    """
    if isinstance(arr, np.ndarray):
        return np.array
    if is_tf_available() and is_tf_tensor(arr):
        import tensorflow as tf

        return tf.convert_to_tensor
    if is_torch_available() and is_torch_tensor(arr):
        import torch

        return torch.tensor
    if is_flax_available() and is_jax_tensor(arr):
        import jax.numpy as jnp

        return jnp.array
    raise ValueError(f"Cannot convert arrays of type {type(arr)}")

When removing the return row, the script works:

def get_numpy_to_framework_fn(arr) -> Callable:
    """
    Returns a function that converts a numpy array to the framework of the input array.

    Args:
        arr (`np.ndarray`): The array to convert.
    """
    if isinstance(arr, np.ndarray):
        return np.array
    if is_tf_available() and is_tf_tensor(arr):
        import tensorflow as tf

        # return tf.convert_to_tensor
    if is_torch_available() and is_torch_tensor(arr):
        import torch

        # return torch.tensor
    if is_flax_available() and is_jax_tensor(arr):
        import jax.numpy as jnp

        # return jnp.array
    raise ValueError(f"Cannot convert arrays of type {type(arr)}")

If moving the import outside the function (global import), the script also works:

import tensorflow as tf
import torch
import jax.numpy as jnp
def get_numpy_to_framework_fn(arr) -> Callable:
    """
    Returns a function that converts a numpy array to the framework of the input array.

    Args:
        arr (`np.ndarray`): The array to convert.
    """
    if isinstance(arr, np.ndarray):
        return np.array
        
    if is_tf_available() and is_tf_tensor(arr):
        return tf.convert_to_tensor

    if is_torch_available() and is_torch_tensor(arr):
        return torch.tensor

    if is_flax_available() and is_jax_tensor(arr):
        return jnp.array

    raise ValueError(f"Cannot convert arrays of type {type(arr)}")

[Rocketknight1]

cc @ArthurZucker @Cyrilvallez

[Cyrilvallez]

Hey, thanks for reporting this issue @xiuqhou! See #36279 for a fix!