Test alphamatte

modules/README.md

@@ -10,6 +10,8 @@ $ cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -D BUILD_opencv_<r
 
 - **aruco**: ArUco and ChArUco Markers -- Augmented reality ArUco marker and "ChARUco" markers where ArUco markers embedded inside the white areas of the checker board.
 
+- **alphamat**: Computer Vision based Alpha Matting -- Given an input image and a trimap, generate an alpha matte.
+
 - **bgsegm**: Background segmentation algorithm combining statistical background image estimation and per-pixel Bayesian segmentation.
 
 - **bioinspired**: Biological Vision -- Biologically inspired vision model: minimize noise and luminance variance, transient event segmentation, high dynamic range tone mapping methods.

modules/alphamat/CMakeLists.txt

@@ -0,0 +1,17 @@
+if(NOT HAVE_EIGEN)
+  set(DISABLE_MSG "Module opencv_alphamat disabled because the following dependencies are not found:")
+  set(DISABLE_MSG "${DISABLE_MSG} Eigen")
+  message(STATUS ${DISABLE_MSG})
+  ocv_module_disable(alphamat)
+endif()
+
+cmake_minimum_required (VERSION 3.0)
+project (myproject)
+set (CMAKE_CXX_STANDARD 11)
+
+ocv_define_module(alphamat opencv_core
+    opencv_imgproc
+    opencv_objdetect
+    opencv_imgcodecs
+    opencv_photo
+)

modules/alphamat/README.md

@@ -0,0 +1,60 @@
+# Designing Effective Inter-Pixel Information Flow for Natural Image Matting:
+
+Alphamatting is the problem of extracting the foreground from an image. Given the input of image and its corresponding trimap, we try to extract the foreground from the background. Following is an example -
+
+Input Image                | Input trimap              | Ouput Alpha matte
+:-------------------------:|:-------------------------:|:-------------------------:
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/net.png" alt="alt text" width="300" height="200"> | <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/trimap/net.png" alt="alt text" width="300" height="200"> | <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_net.png" alt="alt text" width="300" height="200">
+
+This project is implementation of Information-Flow Matting [Yağız Aksoy, Tunç Ozan Aydın, Marc Pollefeys] [1]. It required implementation of some parts of other papers [2,3].
+
+This is a pixel-affinity based alpha matting algorithm which solves a linear system of equations using preconditioned conjugate gradient method. Affinity-based methods operate by propagating opacity information from known opacity regions(K) into unknown opacity regions(U) using a variety of affinity definitions mentioned as -
+* Color mixture information flow - Opacity transitions in a matte occur as a result of the original colors in the image getting mixed with each other due to transparency or intricate parts of an object. They make use of this fact by representing each pixel in U as a mixture of similarly-colored pixels and the difference is the energy term ECM,  which is to be reduced. This is coded in **cm.hpp**
+* K-to-U information flow - Connections from every pixel in U to both F(foreground pixels) and B(background pixels) are made to facilitate direct information flow from known-opacity regions to even the most remote opacity-transition regions in the image. This is coded in **KtoU.hpp**
+* Intra U information flow - They distribute the information inside U effectively by encouraging pixels with similar colors inside U to have similar opacity. This is coded in **intraU.hpp**
+* Local information flow - Spatial connectivity is one of the main cues for information flow which is achieved by connecting each pixel in U to its immediate neighbors to ensure spatially smooth mattes. This is coded in **local_info.hpp**
+
+Using these information flow, energy/error(E) is obtained as a weighted local composite of E<sub>CM</sub>, E<sub>KU</sub>(K-to-U information flow), E<sub>UU</sub>(Intra U information flow), E<sub>L</sub>(Local information flow).
+E represents the deviation of unknown pixels opacity or colour from what we predict it to be using other pixels. So, the algorithm aims at minimizing this error. This is coded in **alphac.cpp**
+
+Pre-processing and post-processing is implemented in **trimming.hpp**
+
+Compile the module using -
+```
+cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -Dopencv_alphamat=ON <opencv_source_dir>
+make
+```
+
+## Results
+
+Results for input_lowres are available here -
+https://docs.google.com/document/d/1BJG4633_U5K-Z0QLp3RTi43q25NI0hrTw-Q4w_85NrA/edit?usp=sharing
+
+Input Image             | Ouput Alpha matte
+:-------------------------:|:-------------------------:
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/net.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_net.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/doll.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_doll.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/donkey.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_donkey.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/elephant.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_elephant.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/pineapple.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_pineapple.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/plant.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_plant.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/plasticbag.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_plasticbag.png" alt="alt text" width="200" height="155">
+<img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/img/troll.png" alt="alt text" width="200" height="155">  |  <img src="https://github.com/muskaankularia/opencv_contrib/blob/alphamat/modules/alphamat/Result/result_troll.png" alt="alt text" width="200" height="155">
+
+Average time taken to compute the different flows is 40s, but solving of linear equations using preconditioned conjugate gradient method takes another 2-3 min, which can be lessened by allowing lesser iterations.
+
+## TO DO
+
+* Results need to be improved by extensively comparing each flow's matrix with yaksoy MATLAB implementation [4].
+* Runtime needs improvement.
+* Third part library(Eigen, nanoflann) dependencies can be removed.
+
+## References
+
+[1] Yagiz Aksoy, Tunc Ozan Aydin, Marc Pollefeys, "Designing Effective Inter-Pixel Information Flow for Natural Image Matting", CVPR, 2017. [[link](http://people.inf.ethz.ch/aksoyy/ifm/)]
+
+[2] Roweis, Sam T., and Lawrence K. Saul. "Nonlinear dimensionality reduction by locally linear embedding." science 290.5500 (2000): 2323-2326.[[link](https://science.sciencemag.org/content/290/5500/2323)]
+
+[3] Ehsan Shahrian, Deepu Rajan, Brian Price, Scott Cohen, "Improving Image Matting using Comprehensive Sampling Sets", CVPR 2013 [[paper](http://www.cv-foundation.org/openaccess/content_cvpr_2013/papers/Shahrian_Improving_Image_Matting_2013_CVPR_paper.pdf)]
+
+[4] Affinity Based Matting Toolbox by Yagiz Aksoy[[link](https://github.com/yaksoy/AffinityBasedMattingToolbox)]

modules/alphamat/doc/alphamat.bib

@@ -0,0 +1,26 @@
+@inproceedings{aksoy2017designing,
+  title={Designing effective inter-pixel information flow for natural image matting},
+  author={Aksoy, Yagiz and Ozan Aydin, Tunc and Pollefeys, Marc},
+  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
+  pages={29--37},
+  year={2017}
+}
+
+@article{roweis2000nonlinear,
+  title={Nonlinear dimensionality reduction by locally linear embedding},
+  author={Roweis, Sam T and Saul, Lawrence K},
+  journal={science},
+  volume={290},
+  number={5500},
+  pages={2323--2326},
+  year={2000},
+  publisher={American Association for the Advancement of Science}
+}
+
+@inproceedings{shahrian2013improving,
+  title={Improving image matting using comprehensive sampling sets},
+  author={Shahrian, Ehsan and Rajan, Deepu and Price, Brian and Cohen, Scott},
+  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
+  pages={636--643},
+  year={2013}
+}

modules/alphamat/include/opencv2/infoflow.hpp

@@ -0,0 +1,31 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#ifndef _OPENCV_ALPHAMAT_INFOFLOW_HPP_
+#define _OPENCV_ALPHAMAT_INFOFLOW_HPP_
+
+/** Information Flow algorithm implementaton for alphamatting
+
+This module contains functionality for extracting the
+The following four models are implemented:
+
+- EDSR <https://arxiv.org/abs/1707.02921>
+- ESPCN <https://arxiv.org/abs/1609.05158>
+- FSRCNN <https://arxiv.org/abs/1608.00367>
+- LapSRN <https://arxiv.org/abs/1710.01992>
+
+*/
+#include <Eigen/Sparse>
+using namespace Eigen;
+
+namespace cv{ namespace alphamat{
+
+    void solve(SparseMatrix<double> Wcm,SparseMatrix<double> Wuu,SparseMatrix<double> Wl,SparseMatrix<double> Dcm,
+            SparseMatrix<double> Duu,SparseMatrix<double> Dl,SparseMatrix<double> H,SparseMatrix<double> T,
+            Mat& ak, Mat& wf, bool useKU, Mat& alpha);
+
+    CV_EXPORTS_W void infoFlow(Mat& image, Mat& tmap, Mat& result, bool useKU, bool trim);
+
+}}
+#endif

modules/alphamat/samples/information_flow_matting.cpp

@@ -0,0 +1,81 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#include <iostream>
+#include "opencv2/highgui.hpp"
+#include <opencv2/core/base.hpp>
+#include <opencv2/core/utility.hpp>
+#include <opencv2/imgproc.hpp>
+#include <opencv2/infoflow.hpp>
+
+using namespace std;
+using namespace cv;
+using namespace cv::alphamat;
+
+const char* keys =
+{
+    "{img || input image name}"
+    "{tri || input trimap image name}"
+    "{out || output image name}"
+};
+
+int main(int argc, char *argv[])
+{
+    bool show_help = (argc == 1);
+    show_help = show_help || (argc == 2 && string(argv[1]) == "--help");
+    show_help = show_help || (argc == 2 && string(argv[1]) == "-h");
+
+    if (show_help)
+    {
+        printf("\nThis sample demonstrates Information Flow alpha matting\n"
+               "Call:\n"
+               "    alphamat_information_flow -img=<string> -tri=<string> [-out=<string>]\n\n");
+        return 0;
+    }
+
+    CommandLineParser parser(argc, argv, keys);
+    if (!parser.check())
+    {
+        parser.printErrors();
+        return -1;
+    }
+
+    string img_path = parser.get<std::string>("img");
+    string trimap_path = parser.get<std::string>("tri");
+    string result_path = parser.get<std::string>("out");
+
+    Mat image, tmap;
+
+    image = imread(img_path, IMREAD_COLOR);   // Read the input image file
+    if (image.empty())
+    {
+        printf("Cannot read image file: %s\n", img_path.c_str());
+        return -1;
+    }
+
+    tmap = imread(trimap_path, IMREAD_GRAYSCALE);
+    if (tmap.empty())
+    {
+        printf("Cannot read trimap file: %s\n", trimap_path.c_str());
+        return -1;
+    }
+
+    Mat result;
+    infoFlow(image, tmap, result, false, true);
+
+    if (result_path.empty())
+    {
+        namedWindow("result alpha matte", WINDOW_NORMAL);
+        imshow("result alpha matte", result);
+        waitKey(0);
+    }
+    else
+    {
+        imwrite(result_path, result);
+    }
+
+    imshow("Result Matte", result);
+    return 0;
+
+}

modules/alphamat/src/KDTreeVectorOfVectorsAdaptor.h

@@ -0,0 +1,116 @@
+/***********************************************************************
+ * Software License Agreement (BSD License)
+ *
+ * Copyright 2011-16 Jose Luis Blanco ([email protected]).
+ *   All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+ * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+ * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
+ * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *************************************************************************/
+
+#pragma once
+
+#include "nanoflann.hpp"
+
+#include <vector>
+
+// ===== This example shows how to use nanoflann with these types of containers: =======
+//typedef std::vector<std::vector<double> > my_vector_of_vectors_t;
+//typedef std::vector<Eigen::VectorXd> my_vector_of_vectors_t;   // This requires #include <Eigen/Dense>
+// =====================================================================================
+
+
+/** A simple vector-of-vectors adaptor for nanoflann, without duplicating the storage.
+  *  The i'th vector represents a point in the state space.
+  *
+  *  \tparam DIM If set to >0, it specifies a compile-time fixed dimensionality for the points in the data set, allowing more compiler optimizations.
+  *  \tparam num_t The type of the point coordinates (typically, double or float).
+  *  \tparam Distance The distance metric to use: nanoflann::metric_L1, nanoflann::metric_L2, nanoflann::metric_L2_Simple, etc.
+  *  \tparam IndexType The type for indices in the KD-tree index (typically, size_t of int)
+  */
+template <class VectorOfVectorsType, typename num_t = double, int DIM = -1, class Distance = nanoflann::metric_L2, typename IndexType = size_t>
+struct KDTreeVectorOfVectorsAdaptor
+{
+  typedef KDTreeVectorOfVectorsAdaptor<VectorOfVectorsType, num_t, DIM,Distance> self_t;
+  typedef typename Distance::template traits<num_t, self_t>::distance_t metric_t;
+  typedef nanoflann::KDTreeSingleIndexAdaptor< metric_t, self_t, DIM, IndexType>  index_t;
+
+  index_t* index; //! The kd-tree index for the user to call its methods as usual with any other FLANN index.
+
+  /// Constructor: takes a const ref to the vector of vectors object with the data points
+  KDTreeVectorOfVectorsAdaptor(const size_t /* dimensionality */, const VectorOfVectorsType &mat, const int leaf_max_size = 10) : m_data(mat)
+  {
+    assert(mat.size() != 0 && mat[0].size() != 0);
+    const size_t dims = mat[0].size();
+    if (DIM>0 && static_cast<int>(dims) != DIM)
+      throw std::runtime_error("Data set dimensionality does not match the 'DIM' template argument");
+    index = new index_t( static_cast<int>(dims), *this /* adaptor */, nanoflann::KDTreeSingleIndexAdaptorParams(leaf_max_size ) );
+    index->buildIndex();
+  }
+
+  ~KDTreeVectorOfVectorsAdaptor() {
+    delete index;
+  }
+
+  const VectorOfVectorsType &m_data;
+
+  /** Query for the \a num_closest closest points to a given point (entered as query_point[0:dim-1]).
+    *  Note that this is a short-cut method for index->findNeighbors().
+    *  The user can also call index->... methods as desired.
+    * \note nChecks_IGNORED is ignored but kept for compatibility with the original FLANN interface.
+    */
+  inline void query(const num_t *query_point, const size_t num_closest, IndexType *out_indices, num_t *out_distances_sq, const int nChecks_IGNORED = 10) const
+  {
+    nanoflann::KNNResultSet<num_t, IndexType> resultSet(num_closest);
+    resultSet.init(out_indices, out_distances_sq);
+    index->findNeighbors(resultSet, query_point, nanoflann::SearchParams());
+  }
+
+  /** @name Interface expected by KDTreeSingleIndexAdaptor
+    * @{ */
+
+  const self_t & derived() const {
+    return *this;
+  }
+  self_t & derived()       {
+    return *this;
+  }
+
+  // Must return the number of data points
+  inline size_t kdtree_get_point_count() const {
+    return m_data.size();
+  }
+
+  // Returns the dim'th component of the idx'th point in the class:
+  inline num_t kdtree_get_pt(const size_t idx, const size_t dim) const {
+    return m_data[idx][dim];
+  }
+
+  // Optional bounding-box computation: return false to default to a standard bbox computation loop.
+  // Return true if the BBOX was already computed by the class and returned in "bb" so it can be avoided to redo it again.
+  // Look at bb.size() to find out the expected dimensionality (e.g. 2 or 3 for point clouds)
+  template <class BBOX>
+  bool kdtree_get_bbox(BBOX & /*bb*/) const {
+    return false;
+  }
+
+  /** @} */
+};  // end of KDTreeVectorOfVectorsAdaptor