Stable-Scan: Retexture 3D Point Clouds Using Stable Diffusion

Stable-Scan leverages the power of Stable Diffusion to enrich your 3D Point Clouds with new textures.

Installation

Clone Repository

First, clone the repository to your local machine:

git clone https://github.com/memben/stable-scan.git

Environment Setup

1. Create a Virtual Environment: Navigate to the project directory and create a virtual environment.

   cd stable-scan
   python3 -m venv .venv  # On Linux or macOS
   py -m venv .venv  # On Windows

2. Activate the Virtual Environment:

   source .venv/bin/activate  # On Linux or macOS
   .\.venv\Scripts\Activate  # On Windows

3. Install Required Packages: Install all the dependencies listed in the requirements.txt file.

   pip install -r requirements.txt

LAS File Creation

The application currently supports only .las files for 3D Point Clouds. You can create .las files using various applications. One such application is ScanKit, also available in the App Store.

Note

For larger scans, ScanKit will produce multiple .las files.

python stablescan.py debug path/to/file1.las path/to/file2.las

Stable Diffusion API Setup

Stable-Scan relies on the Stable Diffusion WebUI API. To set it up:

1. Follow the instructions at the Stable Diffusion WebUI GitHub repo.
2. Install sd-webui-controlnet as an extension.
3. Download the depth model (control_v11f1p_sd15_depth [cfd03158], to use newer versions, update webui_api.py).
4. Start the server:

./webui.sh --api

If you're running the server remotely, use the --share flag.

Usage

Activate your virtual environment and navigate to the project directory.

For CLI options:

python stablescan.py

For a quick demo:

python stablescan.py control path/to/file.las

For full control (recommended), use the debug mode:

python stablescan.py debug path/to/file.las --webui-api SERVER_URL

Workflow:

1. Navigate to the view you wish to retexture.
2. Press r, validate the preview images and if statisfied continue by giving a prompt in CLI.
3. Press o to save the state.
4. Optional: Remove untextured points by pressing x.
5. Load the saved state by pressing l.
6. Navigate to new positions and retexture them.

Known Issues

• Style transfer is currently problematic (See Issue).
• Projection into the point cloud has some known limitations (See Issue).

For more information and updates, visit the Issues section.

We hope you find Stable-Scan useful for your 3D Point Cloud texturing needs! Feel free to contribute and report issues.

How it is build

For the retexturing we use StableDiffusion + ControlNet, using the depth model we can create fitting textures for a scene and project them into the point cloud. The concept looks like this: