Image inpainting-ZhiMap思维导图

Image inpainting

IDEA

main idea
- prolong the isophote lines arriving at $\partial \Omega$
- while maintaining the angle of "arrival"
- prevent prolongation lines from crossing each other

how experts inpaint
- global picture -> how to fill in the gap
- structure of $\partial \Omega$ is continued into the gap, contour lines
- fill $\Omega$ with color, matching those of $\partial \Omega$
- paint small details(texture)

learn lesson from expert
- simultaneously and iteratively perform the steps 2 and 3

METHOD

general form
- evolution equation runs iteratively inside $\Omega$
- $\Delta t$ is the rate of improvement, ${I}_t^n(i,j)$ is the update

design the update ${I}_t^n(i,j)$
- smoothly propagate information from $\partial \Omega$ into $\Omega$
- - $L^n(i,j)$ is the propagation information
  - $\Delta L$ measures the change of $L^n(i,j)$
- algorithm converge
  - means $L^n(i,j)$ has been propagated in the direction $N$

express the information $L$
- since we want the propagation to be smooth, $L^n(i,j)$ should be an image smoothness estimator
- use discrete implementation of the Laplacian as estimator $L^n(i,j) := I_{xx}^n(i,j)+I_{yy}^n(i,j)$
  使用拉普拉斯算子代表像素 I(i,j)的信息/平滑程度。L(i,j)的信息等于它的上下左右四个像素之和减去该像素的四倍。

express the direction $N$
- a bad choice: "shrinked version" of $\partial \Omega$
  - by the belief that a propagation normal to the boundary would lead to the continuity of the isophotes at the boundary
  - fail to reconstruct the local image geometry
- a suitable choice: the isophotes directions
  - by the belief that isophotes tend to align with $N$
  - bootstrapping problem: it's easily recover the gray level image from its isophote direction field
  - use a time varying estimation of the isophotes direction field
```
% EXTRACTING ISOPHOTES FROM AN IMAGE
%
% Usage:  [ I, theta ] = isophote( L, alpha );
%
% Argument:   L     -  Luminance of the input image [0 255]
%             alpha -  Threshold of isophotes       [0 1] 
%               
% Returns:    I     -  Magnitude of the isophotes   [0 1]
%             theta -  Angle of the isophotes   [-pi/2 +pi/2] 
% Vahid. K. Alilou
% Department of Computer Engineering
% The University of Semnan
%
% December 2014
function [ I, theta ] = isophote( L, alpha )
    L = double(L)/255; theta=zeros(size(L));
    [Lx,Ly] = gradient(L);
    I = sqrt(Lx.^2+Ly.^2);
    I = I./max(max(I));
    T = I>=alpha;
    theta(T) = atan(Ly(T)./Lx(T));
    I(I<alpha)=0;
end
```
    - gives the direction of largest spatial change
    - gives the smallest spatial change(isophotes direction)
    - progressively achieves the desired continuity at $\partial \Omega$
  - use a diffusion process to ensure a correct evolution of the direction field
    - every few steps, we apply a few iterations of image diffusion
    - periodical curving of lines to avoid them from crossing each other
    - use anisotropic diffusion to avoid losing sharpness
    - k是等照线的曲率，曲率越大diffusion的系数就越大。g的取值限定了diffusion只在待修复区域进行

compute the change $\Delta L$ of this value along $N$

Recapping, we estimate a variation of the smoothness, given by a discretization of the 2D Laplacian in our case, and project this variation into the isophotes direction. This projection is used to update the value of the image inside the region to be inpainted.

DETAIL

preprocessing
- given a being restored image and its mask
- the whole original image undergoes anisotropic diffusion smoothing
- to minimize the influence of noise on the estimation of the direction of the isophotes arriving at $\partial \Omega$

inpainting loop
- only the values inside are modified $\partial Ω$
- discrete implementation
  - first compute the 2D smoothness estimation L
  - then compute $\beta^n$ , the projection of $\Delta L$ onto the direction $N$
  - finally multiply $\beta^n$ by a slope-limited version of the gradient of the image
- Every few iterations, a step of anisotropic diffusion is applied
- repeated until a steady state is achieved

color image
- Color images are considered as a set of three images, and the
  above described technique is applied independently to each one
- use a color model to avoid the appearance of spurious color

Image inpainting

​IDEA

​main idea

​prolong the isophote lines arriving at ∂Ω \partial \Omega∂Ω

​while maintaining the angle of "arrival"

​prevent prolongation lines from crossing each other

​how experts inpaint

​global picture -> how to fill in the gap

​structure of ∂Ω \partial \Omega∂Ω is continued into the gap, contour lines

​fill Ω\OmegaΩ with color, matching those of ∂Ω \partial \Omega∂Ω

​paint small details(texture)

​learn lesson from expert

​simultaneously and iteratively perform the steps 2 and 3

​METHOD

​general form

evolution equation runs ​iteratively inside Ω\OmegaΩ

​ Δt\Delta t Δt is the rate of improvement, Itn(i,j){I}_t^n(i,j)Itn​(i,j) is the update

​design the update Itn(i,j){I}_t^n(i,j)Itn​(i,j)

​smoothly propagate information from ∂Ω \partial \Omega∂Ω into Ω\OmegaΩ

​

​ Ln(i,j)L^n(i,j)Ln(i,j) is the propagation information

​ ΔL\Delta LΔL measures the change of Ln(i,j)L^n(i,j)Ln(i,j)

​algorithm converge

​

​ means Ln(i,j)L^n(i,j)Ln(i,j) has been propagated in the direction NNN

​express the information LLL

​since we want the propagation to be smooth, Ln(i,j)L^n(i,j) Ln(i,j) should be an image smoothness estimator

​use discrete implementation of the Laplacian as estimator Ln(i,j):=Ixxn(i,j)+Iyyn(i,j)L^n(i,j) := I_{xx}^n(i,j)+I_{yy}^n(i,j)Ln(i,j):=Ixxn​(i,j)+Iyyn​(i,j)

​express the direction NNN

​a bad choice: "shrinked version" of ∂Ω \partial \Omega∂Ω

​by the belief that a propagation normal to the boundary would lead to the continuity of the isophotes at the boundary

​

​

​fail to reconstruct the local image geometry

​a suitable choice: the isophotes directions

​by the belief that isophotes tend to align with NNN

bootstrapping problem: it's easily recover the gray level image from its isophote direction field

​use a time varying estimation of the isophotes direction field

​ gives the direction of largest spatial change

​ gives the smallest spatial change(isophotes direction)

​

​ progressively achieves the desired continuity at ∂Ω \partial \Omega∂Ω

​use a diffusion process to ensure a correct evolution of the direction field

​every few steps, we apply a few iterations of image diffusion

​periodical curving of lines to avoid them from crossing each other

​use anisotropic diffusion to avoid losing sharpness

​k是等照线的曲率，曲率越大diffusion的系数就越大。g的取值限定了diffusion只在待修复区域进行

​compute the change ΔL\Delta LΔL of this value along NNN

​Recapping, we estimate a variation of the smoothness, given by a discretization of the 2D Laplacian in our case, and project this variation into the isophotes direction. This projection is used to update the value of the image inside the region to be inpainted.

​DETAIL

​preprocessing

​given a being restored image and its mask

​the whole original image undergoes anisotropic diffusion smoothing

​to minimize the influence of noise on the estimation of the direction of the isophotes arriving at ∂Ω \partial \Omega∂Ω

​inpainting loop

​only the values inside are modified ∂Ω

​discrete implementation

​first compute the 2D smoothness estimation L

​then compute βn\beta^nβn , the projection of ΔL\Delta LΔL onto the direction NNN

finally multiply βn\beta^nβn by a slope-limited version of the gradient of the image

​Every few iterations, a step of anisotropic diffusion is applied

​repeated until a steady state is achieved

​color image

​Color images are considered as a set of three images, and theabove described technique is applied independently to each one

​ use a color model to avoid the appearance of spurious color

IDEA

main idea

prolong the isophote lines arriving at $\partial \Omega$

while maintaining the angle of "arrival"

prevent prolongation lines from crossing each other

how experts inpaint

global picture -> how to fill in the gap

structure of $\partial \Omega$ is continued into the gap, contour lines

fill $\Omega$ with color, matching those of $\partial \Omega$

paint small details(texture)

learn lesson from expert

simultaneously and iteratively perform the steps 2 and 3

METHOD

general form

evolution equation runs iteratively inside $\Omega$

$\Delta t$ is the rate of improvement, ${I}_t^n(i,j)$ is the update

design the update ${I}_t^n(i,j)$

smoothly propagate information from $\partial \Omega$ into $\Omega$

$L^n(i,j)$ is the propagation information

$\Delta L$ measures the change of $L^n(i,j)$

algorithm converge

means $L^n(i,j)$ has been propagated in the direction $N$

express the information $L$

since we want the propagation to be smooth, $L^n(i,j)$ should be an image smoothness estimator

use discrete implementation of the Laplacian as estimator $L^n(i,j) := I_{xx}^n(i,j)+I_{yy}^n(i,j)$

express the direction $N$

a bad choice: "shrinked version" of $\partial \Omega$

by the belief that a propagation normal to the boundary would lead to the continuity of the isophotes at the boundary

fail to reconstruct the local image geometry

a suitable choice: the isophotes directions

by the belief that isophotes tend to align with $N$

use a time varying estimation of the isophotes direction field

gives the direction of largest spatial change

gives the smallest spatial change(isophotes direction)

progressively achieves the desired continuity at $\partial \Omega$

use a diffusion process to ensure a correct evolution of the direction field

every few steps, we apply a few iterations of image diffusion

periodical curving of lines to avoid them from crossing each other

use anisotropic diffusion to avoid losing sharpness

k是等照线的曲率，曲率越大diffusion的系数就越大。g的取值限定了diffusion只在待修复区域进行

compute the change $\Delta L$ of this value along $N$

Recapping, we estimate a variation of the smoothness, given by a discretization of the 2D Laplacian in our case, and project this variation into the isophotes direction. This projection is used to update the value of the image inside the region to be inpainted.

DETAIL

preprocessing

given a being restored image and its mask

the whole original image undergoes anisotropic diffusion smoothing

to minimize the influence of noise on the estimation of the direction of the isophotes arriving at $\partial \Omega$

inpainting loop

only the values inside are modified $\partial Ω$

discrete implementation

first compute the 2D smoothness estimation L

then compute $\beta^n$ , the projection of $\Delta L$ onto the direction $N$

finally multiply $\beta^n$ by a slope-limited version of the gradient of the image

Every few iterations, a step of anisotropic diffusion is applied

repeated until a steady state is achieved

color image

Color images are considered as a set of three images, and the
above described technique is applied independently to each one

use a color model to avoid the appearance of spurious color