Skip to content

Commit 88c4ed0

Browse files
authored
Computer Vision based Alpha Matting (#2306)
* Computer Vision based Alpha Matting Code alpha matting code This is a combination of 3 commits. removed whitespaces addressed issues raised in the PR removed whitespaces * removed global variable * incorporated changes suggested by second round of review * updated build instructions * changed to OutputArray * removed whitespaces * alphamat: fix bugs triggered by assertions of Debug builds * alphamat: fix documentation * alphamat: coding style fixes - get rid of std::cout - remove clock_t - drop unnecessary cast: float pix = tmap.at<uchar>(i, j); - global 'dim' => 'ALPHAMAT_DIM' - fix sample command line handling * alphamat: apply clang-format * clang-format fixups
1 parent bf0075a commit 88c4ed0

20 files changed

+3073
-0
lines changed

modules/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ $ cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -D BUILD_opencv_<r
1010

1111
- **aruco**: ArUco and ChArUco Markers -- Augmented reality ArUco marker and "ChARUco" markers where ArUco markers embedded inside the white areas of the checker board.
1212

13+
- **alphamat**: Computer Vision based Alpha Matting -- Given an input image and a trimap, generate an alpha matte.
14+
1315
- **bgsegm**: Background segmentation algorithm combining statistical background image estimation and per-pixel Bayesian segmentation.
1416

1517
- **bioinspired**: Biological Vision -- Biologically inspired vision model: minimize noise and luminance variance, transient event segmentation, high dynamic range tone mapping methods.

modules/alphamat/CMakeLists.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
if(NOT HAVE_EIGEN)
2+
message(STATUS "Module opencv_alphamat disabled because the following dependencies are not found: Eigen")
3+
ocv_module_disable(alphamat)
4+
endif()
5+
6+
ocv_define_module(alphamat
7+
opencv_core
8+
opencv_imgproc
9+
)

modules/alphamat/README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Computer Vision based Alpha Matting
2+
3+
This project was part of the Google Summer of Code 2019.
4+
5+
####Student: Muskaan Kularia
6+
####Mentor: Sunita Nayak
7+
***
8+
Alphamatting is the problem of extracting the foreground from an image. Given the input of an image and its corresponding trimap, we try to extract the foreground from the background.
9+
10+
This project is implementation of "[[Designing Effective Inter-Pixel Information Flow for Natural Image Matting](http://people.inf.ethz.ch/aksoyy/ifm/)]" by Yağız Aksoy, Tunç Ozan Aydın and Marc Pollefeys[1]. It required implementation of parts of other papers [2,3,4].
11+
12+
13+
## References
14+
15+
[1] Yagiz Aksoy, Tunc Ozan Aydin, Marc Pollefeys, "[Designing Effective Inter-Pixel Information Flow for Natural Image Matting](http://people.inf.ethz.ch/aksoyy/ifm/)", CVPR, 2017.
16+
17+
[2] Roweis, Sam T., and Lawrence K. Saul. "[Nonlinear dimensionality reduction by locally linear embedding](https://science.sciencemag.org/content/290/5500/2323)" Science 290.5500 (2000): 2323-2326.
18+
19+
[3] Anat Levin, Dani Lischinski, Yair Weiss, "[A Closed Form Solution to Natural Image Matting](https://www.researchgate.net/publication/5764820_A_Closed-Form_Solution_to_Natural_Image_Matting)", IEEE TPAMI, 2008.
20+
21+
[4] Qifeng Chen, Dingzeyu Li, Chi-Keung Tang, "[KNN Matting](http://dingzeyu.li/files/knn-matting-tpami.pdf)", IEEE TPAMI, 2013.
22+
23+
[5] Yagiz Aksoy, "[Affinity Based Matting Toolbox](https://github.com/yaksoy/AffinityBasedMattingToolbox)".

modules/alphamat/doc/alphamat.bib

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
@inproceedings{aksoy2017designing,
2+
title={Designing effective inter-pixel information flow for natural image matting},
3+
author={Aksoy, Yagiz and Ozan Aydin, Tunc and Pollefeys, Marc},
4+
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
5+
pages={29--37},
6+
year={2017}
7+
}
8+
9+
@article{roweis2000nonlinear,
10+
title={Nonlinear dimensionality reduction by locally linear embedding},
11+
author={Roweis, Sam T and Saul, Lawrence K},
12+
journal={science},
13+
volume={290},
14+
number={5500},
15+
pages={2323--2326},
16+
year={2000},
17+
publisher={American Association for the Advancement of Science}
18+
}
19+
20+
@inproceedings{shahrian2013improving,
21+
title={Improving image matting using comprehensive sampling sets},
22+
author={Shahrian, Ehsan and Rajan, Deepu and Price, Brian and Cohen, Scott},
23+
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
24+
pages={636--643},
25+
year={2013}
26+
}
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// This file is part of OpenCV project.
2+
// It is subject to the license terms in the LICENSE file found in the top-level directory
3+
// of this distribution and at http://opencv.org/license.html.
4+
5+
/** Information Flow algorithm implementaton for alphamatting */
6+
7+
#ifndef _OPENCV_ALPHAMAT_HPP_
8+
#define _OPENCV_ALPHAMAT_HPP_
9+
10+
/**
11+
* @defgroup alphamat Alpha Matting
12+
* This module is dedicated to compute alpha matting of images, given the input image and an input trimap.
13+
* The samples directory includes easy examples of how to use the module.
14+
*/
15+
16+
namespace cv { namespace alphamat {
17+
//! @addtogroup alphamat
18+
//! @{
19+
20+
/**
21+
* The implementation is based on Designing Effective Inter-Pixel Information Flow for Natural Image Matting by Yağız Aksoy, Tunç Ozan Aydın and Marc Pollefeys, CVPR 2019.
22+
*
23+
* This module has been originally developed by Muskaan Kularia and Sunita Nayak as a project
24+
* for Google Summer of Code 2019 (GSoC 19).
25+
*
26+
*/
27+
CV_EXPORTS_W void infoFlow(InputArray image, InputArray tmap, OutputArray result);
28+
29+
//! @}
30+
}} // namespace
31+
32+
#endif
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
// This file is part of OpenCV project.
2+
// It is subject to the license terms in the LICENSE file found in the top-level directory
3+
// of this distribution and at http://opencv.org/license.html.
4+
5+
#include <iostream>
6+
#include "opencv2/highgui.hpp"
7+
#include <opencv2/core.hpp>
8+
#include <opencv2/imgproc.hpp>
9+
#include <opencv2/alphamat.hpp>
10+
11+
using namespace std;
12+
using namespace cv;
13+
using namespace cv::alphamat;
14+
15+
const char* keys =
16+
"{img || input image name}"
17+
"{tri || input trimap image name}"
18+
"{out || output image name}"
19+
"{help h || print help message}"
20+
;
21+
22+
int main(int argc, char* argv[])
23+
{
24+
CommandLineParser parser(argc, argv, keys);
25+
parser.about("This sample demonstrates Information Flow Alpha Matting");
26+
27+
if (parser.has("help"))
28+
{
29+
parser.printMessage();
30+
return 0;
31+
}
32+
33+
string img_path = parser.get<std::string>("img");
34+
string trimap_path = parser.get<std::string>("tri");
35+
string result_path = parser.get<std::string>("out");
36+
37+
if (!parser.check()
38+
|| img_path.empty() || trimap_path.empty())
39+
{
40+
parser.printMessage();
41+
parser.printErrors();
42+
return 1;
43+
}
44+
45+
Mat image, tmap;
46+
47+
image = imread(img_path, IMREAD_COLOR); // Read the input image file
48+
if (image.empty())
49+
{
50+
printf("Cannot read image file: '%s'\n", img_path.c_str());
51+
return 1;
52+
}
53+
54+
tmap = imread(trimap_path, IMREAD_GRAYSCALE);
55+
if (tmap.empty())
56+
{
57+
printf("Cannot read trimap file: '%s'\n", trimap_path.c_str());
58+
return 1;
59+
}
60+
61+
Mat result;
62+
infoFlow(image, tmap, result);
63+
64+
if (result_path.empty())
65+
{
66+
namedWindow("result alpha matte", WINDOW_NORMAL);
67+
imshow("result alpha matte", result);
68+
waitKey(0);
69+
}
70+
else
71+
{
72+
imwrite(result_path, result);
73+
printf("Result saved: '%s'\n", result_path.c_str());
74+
}
75+
76+
return 0;
77+
}
Loading
Loading
40.9 KB
Loading
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
/***********************************************************************
2+
* Software License Agreement (BSD License)
3+
*
4+
* Copyright 2011-16 Jose Luis Blanco ([email protected]).
5+
* All rights reserved.
6+
*
7+
* Redistribution and use in source and binary forms, with or without
8+
* modification, are permitted provided that the following conditions
9+
* are met:
10+
*
11+
* 1. Redistributions of source code must retain the above copyright
12+
* notice, this list of conditions and the following disclaimer.
13+
* 2. Redistributions in binary form must reproduce the above copyright
14+
* notice, this list of conditions and the following disclaimer in the
15+
* documentation and/or other materials provided with the distribution.
16+
*
17+
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
18+
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
19+
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
20+
* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
21+
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
22+
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
23+
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
24+
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25+
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
26+
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27+
*************************************************************************/
28+
29+
#pragma once
30+
31+
#include "nanoflann.hpp"
32+
33+
#include <vector>
34+
35+
// ===== This example shows how to use nanoflann with these types of containers: =======
36+
//typedef std::vector<std::vector<double> > my_vector_of_vectors_t;
37+
//typedef std::vector<Eigen::VectorXd> my_vector_of_vectors_t; // This requires #include <Eigen/Dense>
38+
// =====================================================================================
39+
40+
41+
/** A simple vector-of-vectors adaptor for nanoflann, without duplicating the storage.
42+
* The i'th vector represents a point in the state space.
43+
*
44+
* \tparam DIM If set to >0, it specifies a compile-time fixed dimensionality for the points in the data set, allowing more compiler optimizations.
45+
* \tparam num_t The type of the point coordinates (typically, double or float).
46+
* \tparam Distance The distance metric to use: nanoflann::metric_L1, nanoflann::metric_L2, nanoflann::metric_L2_Simple, etc.
47+
* \tparam IndexType The type for indices in the KD-tree index (typically, size_t of int)
48+
*/
49+
template <class VectorOfVectorsType, typename num_t = double, int DIM = -1, class Distance = nanoflann::metric_L2, typename IndexType = size_t>
50+
struct KDTreeVectorOfVectorsAdaptor
51+
{
52+
typedef KDTreeVectorOfVectorsAdaptor<VectorOfVectorsType, num_t, DIM,Distance> self_t;
53+
typedef typename Distance::template traits<num_t, self_t>::distance_t metric_t;
54+
typedef nanoflann::KDTreeSingleIndexAdaptor< metric_t, self_t, DIM, IndexType> index_t;
55+
56+
index_t* index; //! The kd-tree index for the user to call its methods as usual with any other FLANN index.
57+
58+
/// Constructor: takes a const ref to the vector of vectors object with the data points
59+
KDTreeVectorOfVectorsAdaptor(const size_t /* dimensionality */, const VectorOfVectorsType &mat, const int leaf_max_size = 10) : m_data(mat)
60+
{
61+
assert(mat.size() != 0 && mat[0].size() != 0);
62+
const size_t dims = mat[0].size();
63+
if (DIM>0 && static_cast<int>(dims) != DIM)
64+
throw std::runtime_error("Data set dimensionality does not match the 'DIM' template argument");
65+
index = new index_t( static_cast<int>(dims), *this /* adaptor */, nanoflann::KDTreeSingleIndexAdaptorParams(leaf_max_size ) );
66+
index->buildIndex();
67+
}
68+
69+
~KDTreeVectorOfVectorsAdaptor() {
70+
delete index;
71+
}
72+
73+
const VectorOfVectorsType &m_data;
74+
75+
/** Query for the \a num_closest closest points to a given point (entered as query_point[0:dim-1]).
76+
* Note that this is a short-cut method for index->findNeighbors().
77+
* The user can also call index->... methods as desired.
78+
* \note nChecks_IGNORED is ignored but kept for compatibility with the original FLANN interface.
79+
*/
80+
//inline void query(const num_t *query_point, const size_t num_closest, IndexType *out_indices, num_t *out_distances_sq, const int nChecks_IGNORED = 10) const
81+
inline void query(const num_t *query_point, const size_t num_closest, IndexType *out_indices, num_t *out_distances_sq) const
82+
{
83+
nanoflann::KNNResultSet<num_t, IndexType> resultSet(num_closest);
84+
resultSet.init(out_indices, out_distances_sq);
85+
index->findNeighbors(resultSet, query_point, nanoflann::SearchParams());
86+
}
87+
88+
/** @name Interface expected by KDTreeSingleIndexAdaptor
89+
* @{ */
90+
91+
const self_t & derived() const {
92+
return *this;
93+
}
94+
self_t & derived() {
95+
return *this;
96+
}
97+
98+
// Must return the number of data points
99+
inline size_t kdtree_get_point_count() const {
100+
return m_data.size();
101+
}
102+
103+
// Returns the dim'th component of the idx'th point in the class:
104+
inline num_t kdtree_get_pt(const size_t idx, const size_t dim) const {
105+
return m_data[idx][dim];
106+
}
107+
108+
// Optional bounding-box computation: return false to default to a standard bbox computation loop.
109+
// Return true if the BBOX was already computed by the class and returned in "bb" so it can be avoided to redo it again.
110+
// Look at bb.size() to find out the expected dimensionality (e.g. 2 or 3 for point clouds)
111+
template <class BBOX>
112+
bool kdtree_get_bbox(BBOX & /*bb*/) const {
113+
return false;
114+
}
115+
116+
/** @} */
117+
}; // end of KDTreeVectorOfVectorsAdaptor

0 commit comments

Comments
 (0)