WeatherDG

LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation

WeatherDG is a framework for domain generalized semantic segmentation, which can generate realistic and diverse autonomous driving scene images and improve semantic segmentation performance under adverse conditions such as snow, rain, fog, and low-light environments.

Key Features

• Collaborations of Foundation Model: Propose a novel data augmentation framework based on SD and LLM for domain generalization in adverse weather conditions.
• LLM-Agents Utilize collaborations of LLM agents for prompt generation to encourage SD to generate realistic driving-screen samples under adverse weather conditons.
• Sampling strategy Propose a probabilistic sampling strategy for enriching underrepresented objects in adverse weather conditions.

Environment Setup

• Python ≥ 3.8
• PyTorch ≥ 1.10 and torchvision that matches the PYTorch installation. Follow official instruction
• HuggingFace installations: diffusers, transformers, safetensors
• pip install --user -U nltk

Demo Instructions

Image generation

1. Download the Pretrained Model:
   Download and unzip the pretrained model from Google Drive or Tencent Cloud, and change the --sd_path in scripts/gen_data_weather.sh to be the model path.

2. Run the Script: Execute the script using the following command:

   sh scripts/gen_data_weather.sh
   

3. (Alternative) Download generated datasets from this link

Semantic Segmentation Training

You can use the generated dataset for domain adaptive semantic segmentation training. For more details, please refer to MIC and DAFormer

Citation

If you find this work useful, please cite:

@misc{qian2024weatherdg,
  title={WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation}, 
  author={Chenghao Qian and Yuhu Guo and Yuhong Mo and Wenjing Li},
  year={2024},
  eprint={2410.12075},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2410.12075}, 
}