add code and update README

LICENSE

@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
@@ -198,4 +199,4 @@
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

README.md

@@ -1,7 +1,194 @@
-# Objects With Lighting
+# Objects with Lighting
+
+## [Paper](https://arxiv.org/abs/2401.09126)
+
+This repo is the code distribution for the _Objects with Lighting_ dataset.
+It contains the evaluation script (`scripts/evaluation.py`) and the tools used for building the dataset.
+
+If you find the data or code useful please cite
+```bibtex
+@inproceedings{Ummenhofer2024OWL,
+  author       = {Benjamin Ummenhofer and
+                  Sanskar Agrawal and
+                  Rene Sep{\'{u}}lveda and
+                  Yixing Lao and
+                  Kai Zhang and
+                  Tianhang Cheng and
+                  Stephan R. Richter and
+                  Shenlong Wang and
+                  Germ{\'{a}}n Ros},
+  title        = {Objects With Lighting: {A} Real-World Dataset for Evaluating Reconstruction
+                  and Rendering for Object Relighting},
+  booktitle    = {3DV},
+  publisher    = {{IEEE}},
+  year         = {2024}
+}
+```
+
+## Downloads
+
+Please download the dataset from the current release or use the links below for the latest version. Extracting the files in the repository root will create the `dataset` directory.
+
+- [objects-with-lighting-dataset-v1_1.tgz](https://github.com/isl-org/objects-with-lighting/releases/download/v1/objects-with-lighting-dataset-v1_1.tgz)
+- [objects-with-lighting-dataset-v1_2.tgz](https://github.com/isl-org/objects-with-lighting/releases/download/v1/objects-with-lighting-dataset-v1_2.tgz)
+
+```bash
+wget https://github.com/isl-org/objects-with-lighting/releases/download/v1/objects-with-lighting-dataset-v1_{1,2}.tgz
+```
+
+## Directory structure
+
+```
+├─ calibration      # Data files for calibration and generated calibration parameters
+├─ dataset          # Contains the data meant for consumption
+├─ docs             # Images and markdown for documentation
+├─ methods          # Documentation and scripts for the baseline method and other state-of-the-art methods
+├─ scripts          # This dir contains scripts for creating data and evaluation
+├─ utils            # Utility python modules used by all scripts
+```
+
+## Evaluation script
+
+The script `scripts/evaluate.py` can be used to compute the common metrics PSNR, SSIM, LPIPS for predicted images.
+The results will be stored in a json file.
+ - Supported image file formats are `.png`, `.exr`, and `.npy`.
+ - We assume `.exr` and `.npy` files store unclipped linear images, while `.png` stores values after applying the tonemapping as used for the dataset.
+ - For linear images the evaluation script computes the optimal exposure value minimizing the least squares error before computing the error metrics.
+ - The predicted images have to be stored with the same folder structure as the dataset and should be named `pr_image_xxxx.{npy,exr,png}`.
+
+The script can be invoked as 
+```bash
+python scripts/evaluate.py -p path/to/predictions results.json
+```
+
+## Dataset format
+
+```
+├─ dataset
+    ├─ object_name     # The dataset is grouped into objects 
+        ├─ test        # Files in the test dir are meant for evaluation
+            ├─ inputs  # The inputs dir contains all files that are allowed to be used by the methods
+```
+
+Each of the test directories contains the following data.
+
+| File | Description |
+| --- | --- |
+| `inputs/`  | This directory contains all data for reconstructing the object. For a fair evaluation only data inside this folder may be used. |
+| `inputs/image_xxxx.png`  | Image files with 8-bit RGB images after tonemapping. |
+| `inputs/camera_xxxx.txt` | Camera parameters for the corresponding image file. |
+| `inputs/mask_xxxx.png` | An approximate mask for methods that require it. |
+| `inputs/exposure.txt`    | The exposure value that has been used in the tonemapping. |
+| `inputs/object_bounding_box.txt`  | The axis aligned bounding box of the object. This box is not a tight bounding box. |
+| `env.hdr`    | An equirectangular image of the environment where the input images have been taken. This image is provided for debugging purposes and should not be used for reconstruction or evaluation. |
+| `env_512_rotated.hdr` | This environment map is downscaled to 1024x512 and has been rotated with the 'world_to_env' transform for easier usage. This image is provided for debugging purposes and should not be used for reconstruction or evaluation. |
+| `world_to_env.txt` | The 4x4 world to camera transform that transforms a point into the coordinate system of the equirectangular image 'env.hdr'. |
+| `gt_image_xxxx.png` | A ground truth image used in evaluation. |
+| `gt_camera_xxxx.txt` | The corresponding camera parameters for a ground truth image. |
+| `gt_mask_xxxx.png` | The mask used for evaluation. Valid pixels are marked with the value 255. |
+| `gt_exposure_xxxx.txt` | The exposure used in the tonemapping of the corresponding ground truth image. |
+| `gt_env_xxxx.hdr` | An equirectangular image of the environment where the corresponding ground truth image was taken. |
+| `gt_world_to_env_xxxx.txt` | The 4x4 world to camera transform that transforms a point into the coordinate system of the equirectangular image 'gt_env_xxxx.hdr'. |
+| `gt_env_512_rotated_xxxx.hdr` | This environment map is downscaled to 1024x512 and has been rotated with the 'world_to_env' transform for easier usage. |
+
+### Tone mapping
+
+We generate the tonemapped 8-bit images with the following function.
+
+$$ y = 255 (x 2^\text{exposure})^\gamma $$
+
+We use $\gamma=1/2.2$ for all images in the dataset.
+The exposure values used may differ for input and test images. The exposure values can be found in the corresponding `exposure.txt` files.
+The values $y$ are clipped to the range 0 to 255.
+
+### `.txt` files
+
+#### Camera parameters
+The camera parameters are defined by the intrinsic matrix $K$, and the extrinsics $R,t$.
+We can project a 3D point $X$ to the camera coordinate system with 
+
+$$ x = K(R X + t) $$
+
+Note that $x$ is in homogeneous coordinates.
+The camera parameters are stored in the `*camera_xxx.txt` files in the following format.
+```
+k11 k12 k13
+k21 k22 k23
+k31 k32 k33
+r11 r12 r13
+r21 r22 r23
+r31 r32 r33
+tx  ty  tz
+width height channels
+```
+
+The following snippet can be used to parse the file with numpy.
+```python
+params = np.loadtxt('path/to/camera_xxxx.txt')
+K, R, t, (width, height, channels) = params[:3], params[3:6], params[6], params[7].astype(int)
+```
+
+#### Object bounding box
+The `object_bounding_box.txt` files describe an axis aligned bounding box.
+The format used in the text file is
+```
+xmin xmax ymin ymax zmin zmax
+```
+
+#### World to environment map transforms
+The `*world_to_env*.txt` files describe a transformation from the world coordinate system into the coordinate system of the omnidirectional camera that captures the environment.
+The text file stores a 4x4 transformation matrix and transforms a homogeneous 3D point to the camera coordinate system.
+Usually we make the assumption that the environment is infinitely far away from the object and we are only interested in directions.
+In this case we only the rotational part of the 4x4 matrix in the upper left corner is of interest.
+With $R$ as the rotation and $t$ as the translation the format of the text file is
+```
+r11 r12 r13 tx
+r21 r22 r23 ty
+r31 r32 r33 tz
+0   0   0   1
+```
+
+#### Exposure
+The exposure values are single scalars stored in the `*exposure*.txt` files.
+
+## Coordinate systems
+
+The dataset uses right-handed coordinate systems.
+
+### Cameras
+Cameras look in positive z-direction.
+The intrinsic and extrinsic camera parameters can be used to directly project a 3D point $X$ to image space coordinates.
+
+$$ x = K(R X + t) $$
+
+$x$ is a homogeneous point describing a position in the image.
+
+
+### Images
+The x-axis for images points to the right and the y-axis points down following the memory order.
+The coordinates $(x,y)$ of the top left corner are $(0,0)$.
+The center of the first pixel is at $(0.5, 0,5)$.
+The bottom right corner for an image with witdth $w$ and height $h$ is at $(w, h)$.
+
+
+### Environment maps
+Environment maps are stored as equirectangular images. 
+We use a normalized coordinate system similar to regular images.
+The u-axis points to the right and the v-axis points down following the memory order.
+The coordinates $(u,v)$ of the top left corner are $(0,0)$.
+The bottom right corner is at $(1, 1)$ irrespective of the size of the environment map.
+This corresponds to the texture coordinate convention used by DirectX.
+
+![Equirectangular uv coordinates](docs/images/equirect_coords1.png)
+
+Directions map to the equirectangular image as shown in the image below.
+The direction +Z $(0,0,1)$ maps to the upper border of the environment map and -Z $(0,0,-1)$ to the lower border.
++X $(1,0,0)$ maps to the center and -X $(-1,0,0)$ maps to the vertically centered point on the right and left border.
++Y $(0,1,0)$ and -Y $(0,-1,0)$ map to the uv coordinates $(0.25,0.5)$ and $(0.75,0.5)$ respectively.
+
+![Equirectangular xyz directions](docs/images/equirect_coords2.png)
+
+The following shows the left half of the environment map mapped to a sphere.
+![xyz directions with sphere mapped equirectangular image](docs/images/sphere_directions.png)
 
-This is the repository for the paper _Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting_.
 
-## TODOs
-- [ ] Code for evaluation and dataset creation
-- [ ] Upload dataset

calibration/color/eos90d_iso_1000_to_theta.txt

@@ -0,0 +1,3 @@
+1.395296136706085030e-02 1.296499703681861047e-03 1.873509458958052859e-04
+-3.780099064391110805e-03 1.124387368021347554e-02 9.833983419795452874e-04
+1.974805264306137790e-03 -1.998499422070082866e-04 1.065255577973303613e-02

calibration/color/eos90d_iso_100_to_theta.txt

@@ -0,0 +1,3 @@
+1.537416332420910858e-01 1.417908997397511646e-02 1.341531951197547350e-03
+-4.326455047072624849e-02 1.229526531999921707e-01 7.063342058748769969e-03
+2.217579919984779754e-02 -3.933207846438259089e-03 1.209818695892059953e-01

calibration/color/eos90d_iso_2000_to_theta.txt

@@ -0,0 +1,3 @@
+6.936005635577023598e-03 6.069567868976031427e-04 5.363830447564449600e-05
+-1.918221804478201838e-03 5.645104132183957103e-03 5.525334345413131058e-04
+9.269149543259147701e-04 -2.245523645808016793e-04 5.167599107275024084e-03

calibration/color/eos90d_iso_2500_to_theta.txt

@@ -0,0 +1,3 @@
+4.503985349671151689e-03 4.117778554999114698e-04 4.807029340357996794e-05
+-1.266068589016416121e-03 3.636929855187648677e-03 3.342844793383778143e-04
+5.731625967925495137e-04 -1.283951704963701482e-04 3.421848290720163124e-03

calibration/color/eos90d_iso_250_to_theta.txt

@@ -0,0 +1,3 @@
+6.289920340398877374e-02 5.989769682043516973e-03 8.111168215522625985e-04
+-1.730149418944995632e-02 4.990574939938621779e-02 4.010673489420267523e-03
+9.454023743463390445e-03 -9.412456589485911724e-04 4.852574589143844597e-02

calibration/color/eos90d_iso_5000_to_theta.txt

@@ -0,0 +1,3 @@
+2.095289375202849438e-03 1.991256304950709221e-04 3.179490255994356585e-05
+-5.837361111847496221e-04 1.675642306626312856e-03 1.516999247522994067e-04
+3.008858817500313643e-04 -3.593445281422978199e-05 1.586576156932782439e-03

calibration/color/eos90d_iso_500_to_theta.txt

@@ -0,0 +1,3 @@
+2.807759482776742838e-02 2.513098971642670036e-03 2.823983904470497483e-04
+-7.867633287587591506e-03 2.262510365043172642e-02 1.982277948859361913e-03
+4.033590404060006900e-03 -6.398339957334430979e-04 2.133600414741858298e-02

calibration/color/theta_to_theta_scaled.txt

@@ -0,0 +1,3 @@
+1.000000000000000021e-02 0.000000000000000000e+00 0.000000000000000000e+00
+0.000000000000000000e+00 1.000000000000000021e-02 0.000000000000000000e+00
+0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000021e-02

calibration/intrinsics/eos90d_K.txt

@@ -0,0 +1,3 @@
+5.991745083950964727e+03 0.000000000000000000e+00 3.469006466746472142e+03
+0.000000000000000000e+00 5.996834010370436772e+03 2.378733834317746641e+03
+0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00

calibration/intrinsics/eos90d_dist_coeffs.txt

@@ -0,0 +1,5 @@
+3.862831141118877598e-03
+0.000000000000000000e+00
+0.000000000000000000e+00
+0.000000000000000000e+00
+0.000000000000000000e+00

calibration/intrinsics/eos90d_width_height.txt

@@ -0,0 +1,2 @@
+6.984000000000000000e+03
+4.660000000000000000e+03

calibration/intrinsics/theta_left_K.txt

@@ -0,0 +1,3 @@
+1.128307246143313478e+03 0.000000000000000000e+00 1.834778206370225689e+03
+0.000000000000000000e+00 1.129750272269983270e+03 1.828309860267060230e+03
+0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00

calibration/intrinsics/theta_left_dist_coeffs.txt

@@ -0,0 +1,4 @@
+6.568549355559186176e-02
+-3.291644759792560632e-02
+0.000000000000000000e+00
+0.000000000000000000e+00

calibration/intrinsics/theta_left_rig_extrinsics.txt

@@ -0,0 +1,4 @@
+1.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 -2.111996213286720849e-03
+0.000000000000000000e+00 1.000000000000000000e+00 0.000000000000000000e+00 -1.910690494141166072e-04
+0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00 -3.791526405692250117e-03
+0.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00

calibration/intrinsics/theta_left_width_height.txt

@@ -0,0 +1,2 @@
+3.648000000000000000e+03
+3.648000000000000000e+03

calibration/intrinsics/theta_right_K.txt

@@ -0,0 +1,3 @@
+1.189281467580679418e+03 0.000000000000000000e+00 1.840371839991512843e+03
+0.000000000000000000e+00 1.183950873150864481e+03 1.837103569984130900e+03
+0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00

calibration/intrinsics/theta_right_dist_coeffs.txt

@@ -0,0 +1,4 @@
+-1.951546205446428456e-02
+0.000000000000000000e+00
+0.000000000000000000e+00
+0.000000000000000000e+00

calibration/intrinsics/theta_right_rig_extrinsics.txt

@@ -0,0 +1,4 @@
+-9.991646303524051032e-01 -2.232118913805105753e-03 4.080513568041191363e-02 -1.955944655173208913e-03
+-2.638252434545347840e-03 9.999474950030435849e-01 -9.901861503296228995e-03 1.479438688275491742e-04
+-4.078089107454387074e-02 -1.000124403729231969e-02 -9.991180581096875679e-01 -3.876222515441903483e-03
+0.000000000000000000e+00 0.000000000000000000e+00 0.000000000000000000e+00 1.000000000000000000e+00

calibration/intrinsics/theta_right_width_height.txt

@@ -0,0 +1,2 @@
+3.648000000000000000e+03
+3.648000000000000000e+03

calibration/ndfilter/fstop_reduction.txt

@@ -0,0 +1 @@
+1.899058519806426704e+00