Digital Image Processing

Image Restoration

Matrix Formulation

21/11/15

Duong Anh Duc - Digital Image Processing

1

Matrix Formulation

of Image Restoration Problem

1-D Case:

We will consider the 1-D version first, for simplicity:

g(m) = f(m)*h(m) + (m)

We will assume that the arrays f and h have been zeropadded to be of size M, where M length(f) + length(h) 1.

Henceforth, we will not explicitly mention the zero-padding.

The degradation equation:

can be written in matrix-vector form as follows:

g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

2

Matrix Formulation

of Image Restoration Problem

H

H1

h0

h1

h2

hM 1

h 1

h0

h1

hM 2

h0

h1

h2

hM 1

21/11/15

h 2

h 1

h0

hM 3

0

h0

h1

hM 2

h M 1

h M 2

h M 3

h0

0

0

h0

hM 3

Duong Anh Duc - Digital Image Processing

0

0

0

h0

3

Matrix Formulation

of Image Restoration Problem

However, since the arrays f and h are zeropadded, we can equivalently set:

H

H2

h0

h1

h2

hM 1

hM 1

h0

h1

hM 2

hM 2

hM 1

h0

hM 3

h1

h2

h3

h0

Notice that the (second) matrix H is circulant; i.e.,

each row of H is a circular shift of the previous

row.

21/11/15

Duong Anh Duc - Digital Image Processing

4

Matrix Formulation

of Image Restoration Problem

Example:

A

= length of array f = 3

B = length of array h = 2

M

A + B – 1 = 4, say M = 4.

f

21/11/15

f 0

f 1

f 2

0

h

h0

h1

0

0

Duong Anh Duc - Digital Image Processing

5

Matrix Formulation

of Image Restoration Problem

H1

21/11/15

h0

h1

h2

h3

h 1

h0

h1

h2

h0

h1

h2

h3

0

h0

h1

h2

h 2

h 1

h0

h1

0

0

h0

h1

h

h

h

h

0

0

0

h0

3

2

1

0

h0

h1

0

0

Duong Anh Duc - Digital Image Processing

0

h0

h1

0

0

0

h0

h1

0

0

0

h0

6

Matrix Formulation

of Image Restoration Problem

H2

h0

h1

h2

h3

hM 1

h0

h1

h2

h0

h1

h2

h3

h3

h0

h1

h2

h2

h3

h0

h1

hM 2

hM 1

h0

h1

h1

h2

h3

h0

hM 3

hM 2

hM 1

h0

h0

h1

0

0

0

h0

h1

0

0

0

h0

h1

h1

0

0

h0

Notice that H1f = H2f. Indeed

21/11/15

Duong Anh Duc - Digital Image Processing

7

Matrix Formulation

of Image Restoration Problem

H1f

h0

h1

0

0

0

h0

h1

0

0

0

h0

h1

0

0

0

h0

f 0

f 1

f 2

0

h0

0

0

h1 f 0

h1 h 0

0

0

f 1

H 2f

0

h1 h 0

0

f 2

0

0

h1 h 0

0

Henceforth, we will use H = H2 , so that we can apply

properties of circulant matrices to H.

21/11/15

Duong Anh Duc - Digital Image Processing

8

Matrix Formulation

of Image Restoration Problem

2-D Case:

Suppose g, f, h, are M N arrays (after zeropadding). The degradation equation can be

written in matrix-vector format as follows:

g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

9

Matrix Formulation

of Image Restoration Problem

g

g 0,0

f 0,0

0,0

g 0, N 1

g 1,0

f 0, N 1

f 1,0

0, N 1

1,0

g 1, N 1

f 1, N 1

1, N 1

f

n

g M 1,0

f M 1,0

M 1,0

g M 1, N 1

f M 1, N 1

M 1, N 1

21/11/15

MN 1

MN 1

Duong Anh Duc - Digital Image Processing

MN 1

10

Matrix Formulation

of Image Restoration Problem

H

H0

H M1

H M 2

H1

H1

H0

HM

H2

H2

H1

H0

HM

1

HM

2

HM

1

H3

3

H0

MN MN

Note that H is a MN MN block-circulant

matrix with M M blocks.

21/11/15

Duong Anh Duc - Digital Image Processing

11

Matrix Formulation

of Image Restoration Problem

Each block Hj is itself an N N circulant matrix.

Indeed, the matrix Hj is a circulant matrix formed

from the j-th row of array h(m,n):

Hj

21/11/15

h j ,0

h j, N 1

h j, N 2

h j ,1

h j ,1

h j ,2

h j ,0

h j ,1

h j, N 1

h j ,0

h j ,2

h j ,3

h j, N 1

h j, N 2

h j, N 3

h j ,0

Duong Anh Duc - Digital Image Processing

12

Matrix Formulation

of Image Restoration Problem

Given the degradation equation:

g = Hf + n,

our objective is to recover f from observation g.

We will assume that the array h(m,n) (usually

referred to as the blurring function) and statistics of

the noise (m,n) are known. The problem

becomes very complicated if array h(m,n) is

unknown and this case is usually referred to as

blind restoration or blind deconvolution.

21/11/15

Duong Anh Duc - Digital Image Processing

13

Matrix Formulation

of Image Restoration Problem

Notice that, even when there is no noise; i.e.

(m,n) = 0, or the values of (m,n) were

exactly known, and matrix H is invertible,

computing

^f = H-1(g-n)

directly would not be practical.

Example: Suppose M = N = 256. Therefore

MN = 65536 and H would be a 65536 by

65536 matrix to be inverted!

21/11/15

Duong Anh Duc - Digital Image Processing

14

Matrix Formulation

of Image Restoration Problem

Naturally, direct inversion of H would not be feasible.

But H has several useful properties; in particular:

H is block circulant.

H is usually sparse (has very few non-zero entries).

We will exploit these properties to obtain ˆf more efficiently.

In particular, we will derive the theoretical solutions to the

restoration problem using matrix algebra. However, when it

comes to implementing the solution, we can resort to the

Fourier domain, thanks to the properties of circulant

matrices.

21/11/15

Duong Anh Duc - Digital Image Processing

15

Constrained least squares

filtering (restoration)

Recall that the knowledge of blur function

h(m, n) is essential to obtain a meaningful

solution to the restoration problem.

Often, knowledge of h(m, n) is not perfect

and subject to errors.

One way to alleviate sensitivity of the result

to errors in h(m, n) is to base optimality of

restoration on a measure of smoothness,

such as the second derivative of the image.

21/11/15

Duong Anh Duc - Digital Image Processing

16

Constrained least squares

filtering (restoration)

We will approximate the second derivative

(Laplacian) by a matrix Q. Indeed, we will

first formulate the constrained restoration

problem and obtain its solution in terms of a

general matrix Q.

21/11/15

Duong Anh Duc - Digital Image Processing

17

Constrained least squares

filtering (restoration)

Later different choices of matrix Q will be

considered, each giving rise to a different

restoration filter.

Suppose Q is any matrix (of appropriate

dimension). In constrained image restoration, we

choose ^f to minimize ||Qˆf||2, subject to the

constraint,

||g-Hˆf||2= ||n||2.

(Recall the degradation equation

g=Hˆf +n g-Hˆf = n.)

21/11/15

Duong Anh Duc - Digital Image Processing

18

Constrained least squares

filtering (restoration)

Introduction of matrix Q allows considerable

flexibility in the design of appropriate

restoration filters (we will discuss specific

choices of Q later). So our problem is

formulated as follows:

min ||Qˆf||2

subject to ||g-Hˆf||2= ||n||2 or ||g-Hˆf||2- ||n||2=0

21/11/15

Duong Anh Duc - Digital Image Processing

19

A brief review of matrix

differentiation

Suppose

x

x1

x2

and f(x1,x2) is a function of two variables. Then

f x1 , x2

x

21/11/15

f x1 , x2

x1

f x1 , x2

x2

Duong Anh Duc - Digital Image Processing

20

A brief review of matrix

differentiation

If

f(x1,x2) = ||Ax-b||2 = (Axb)T (Axb)

for some matrix A and some vector b , then

f x1 , x2

x

T

2A Ax b

where superscript T denotes matrix

transpose.

21/11/15

Duong Anh Duc - Digital Image Processing

21

A brief review of matrix

differentiation

Recall from calculus that such a constrained

minimization problem can be solved by means of

Lagrange multipliers. We need to minimize the

augmented objective function J(ˆf):

J(ˆf) = ||Qˆf||2+ (||g-Hˆf||2- ||n||2),

where is a Lagrange multiplier.

We set the derivative of J(ˆf) with respect to ˆf to

zero.

J(ˆf) = 2QT Qˆf-2 HT(g-Hˆf) = 0

(QT Q + HTH)ˆf = HTg

21/11/15

Duong Anh Duc - Digital Image Processing

22

A brief review of matrix

differentiation

Therefore,

ˆf = (QT Q + HTH)-1 HTg

= (QT Q/ + HTH)-1HTg

= ( QT Q + HTH)-1HTg

where = 1/ is chosen to satisfy the

constraint ||g-Hˆf||2= ||n||2.

We will now use the above formulation to

derive a number of restoration filters.

21/11/15

Duong Anh Duc - Digital Image Processing

23

Pseudo-inverse Filtering

The pseudo-inverse filter tries to avoid the

pitfalls of applying an inverse filter in the

presence of noise.

Consider the constrained restoration

solution,

^f = ( QT Q + HTH)-1HTg

with Q = I. This gives,

^f = ( I + HTH)-1HTg

21/11/15

Duong Anh Duc - Digital Image Processing

24

Pseudo-inverse Filtering

It can be implemented in the Fourier domain by

the following equation:

^F(u,v) = R(u,v)G(u,v), where

R u, v

*

H u, v

H u, v

2

1

H u, v

H u, v

H u, v

2

2

The parameter is a constant to be chosen.

Note that = 0 gives us back the inverse filter. For

> 0, the denominator of R(u, v) is strictly positive

and the pseudo-inverse filter is well defined.

21/11/15

Duong Anh Duc - Digital Image Processing

25

Image Restoration

Matrix Formulation

21/11/15

Duong Anh Duc - Digital Image Processing

1

Matrix Formulation

of Image Restoration Problem

1-D Case:

We will consider the 1-D version first, for simplicity:

g(m) = f(m)*h(m) + (m)

We will assume that the arrays f and h have been zeropadded to be of size M, where M length(f) + length(h) 1.

Henceforth, we will not explicitly mention the zero-padding.

The degradation equation:

can be written in matrix-vector form as follows:

g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

2

Matrix Formulation

of Image Restoration Problem

H

H1

h0

h1

h2

hM 1

h 1

h0

h1

hM 2

h0

h1

h2

hM 1

21/11/15

h 2

h 1

h0

hM 3

0

h0

h1

hM 2

h M 1

h M 2

h M 3

h0

0

0

h0

hM 3

Duong Anh Duc - Digital Image Processing

0

0

0

h0

3

Matrix Formulation

of Image Restoration Problem

However, since the arrays f and h are zeropadded, we can equivalently set:

H

H2

h0

h1

h2

hM 1

hM 1

h0

h1

hM 2

hM 2

hM 1

h0

hM 3

h1

h2

h3

h0

Notice that the (second) matrix H is circulant; i.e.,

each row of H is a circular shift of the previous

row.

21/11/15

Duong Anh Duc - Digital Image Processing

4

Matrix Formulation

of Image Restoration Problem

Example:

A

= length of array f = 3

B = length of array h = 2

M

A + B – 1 = 4, say M = 4.

f

21/11/15

f 0

f 1

f 2

0

h

h0

h1

0

0

Duong Anh Duc - Digital Image Processing

5

Matrix Formulation

of Image Restoration Problem

H1

21/11/15

h0

h1

h2

h3

h 1

h0

h1

h2

h0

h1

h2

h3

0

h0

h1

h2

h 2

h 1

h0

h1

0

0

h0

h1

h

h

h

h

0

0

0

h0

3

2

1

0

h0

h1

0

0

Duong Anh Duc - Digital Image Processing

0

h0

h1

0

0

0

h0

h1

0

0

0

h0

6

Matrix Formulation

of Image Restoration Problem

H2

h0

h1

h2

h3

hM 1

h0

h1

h2

h0

h1

h2

h3

h3

h0

h1

h2

h2

h3

h0

h1

hM 2

hM 1

h0

h1

h1

h2

h3

h0

hM 3

hM 2

hM 1

h0

h0

h1

0

0

0

h0

h1

0

0

0

h0

h1

h1

0

0

h0

Notice that H1f = H2f. Indeed

21/11/15

Duong Anh Duc - Digital Image Processing

7

Matrix Formulation

of Image Restoration Problem

H1f

h0

h1

0

0

0

h0

h1

0

0

0

h0

h1

0

0

0

h0

f 0

f 1

f 2

0

h0

0

0

h1 f 0

h1 h 0

0

0

f 1

H 2f

0

h1 h 0

0

f 2

0

0

h1 h 0

0

Henceforth, we will use H = H2 , so that we can apply

properties of circulant matrices to H.

21/11/15

Duong Anh Duc - Digital Image Processing

8

Matrix Formulation

of Image Restoration Problem

2-D Case:

Suppose g, f, h, are M N arrays (after zeropadding). The degradation equation can be

written in matrix-vector format as follows:

g = Hf + n, where

21/11/15

Duong Anh Duc - Digital Image Processing

9

Matrix Formulation

of Image Restoration Problem

g

g 0,0

f 0,0

0,0

g 0, N 1

g 1,0

f 0, N 1

f 1,0

0, N 1

1,0

g 1, N 1

f 1, N 1

1, N 1

f

n

g M 1,0

f M 1,0

M 1,0

g M 1, N 1

f M 1, N 1

M 1, N 1

21/11/15

MN 1

MN 1

Duong Anh Duc - Digital Image Processing

MN 1

10

Matrix Formulation

of Image Restoration Problem

H

H0

H M1

H M 2

H1

H1

H0

HM

H2

H2

H1

H0

HM

1

HM

2

HM

1

H3

3

H0

MN MN

Note that H is a MN MN block-circulant

matrix with M M blocks.

21/11/15

Duong Anh Duc - Digital Image Processing

11

Matrix Formulation

of Image Restoration Problem

Each block Hj is itself an N N circulant matrix.

Indeed, the matrix Hj is a circulant matrix formed

from the j-th row of array h(m,n):

Hj

21/11/15

h j ,0

h j, N 1

h j, N 2

h j ,1

h j ,1

h j ,2

h j ,0

h j ,1

h j, N 1

h j ,0

h j ,2

h j ,3

h j, N 1

h j, N 2

h j, N 3

h j ,0

Duong Anh Duc - Digital Image Processing

12

Matrix Formulation

of Image Restoration Problem

Given the degradation equation:

g = Hf + n,

our objective is to recover f from observation g.

We will assume that the array h(m,n) (usually

referred to as the blurring function) and statistics of

the noise (m,n) are known. The problem

becomes very complicated if array h(m,n) is

unknown and this case is usually referred to as

blind restoration or blind deconvolution.

21/11/15

Duong Anh Duc - Digital Image Processing

13

Matrix Formulation

of Image Restoration Problem

Notice that, even when there is no noise; i.e.

(m,n) = 0, or the values of (m,n) were

exactly known, and matrix H is invertible,

computing

^f = H-1(g-n)

directly would not be practical.

Example: Suppose M = N = 256. Therefore

MN = 65536 and H would be a 65536 by

65536 matrix to be inverted!

21/11/15

Duong Anh Duc - Digital Image Processing

14

Matrix Formulation

of Image Restoration Problem

Naturally, direct inversion of H would not be feasible.

But H has several useful properties; in particular:

H is block circulant.

H is usually sparse (has very few non-zero entries).

We will exploit these properties to obtain ˆf more efficiently.

In particular, we will derive the theoretical solutions to the

restoration problem using matrix algebra. However, when it

comes to implementing the solution, we can resort to the

Fourier domain, thanks to the properties of circulant

matrices.

21/11/15

Duong Anh Duc - Digital Image Processing

15

Constrained least squares

filtering (restoration)

Recall that the knowledge of blur function

h(m, n) is essential to obtain a meaningful

solution to the restoration problem.

Often, knowledge of h(m, n) is not perfect

and subject to errors.

One way to alleviate sensitivity of the result

to errors in h(m, n) is to base optimality of

restoration on a measure of smoothness,

such as the second derivative of the image.

21/11/15

Duong Anh Duc - Digital Image Processing

16

Constrained least squares

filtering (restoration)

We will approximate the second derivative

(Laplacian) by a matrix Q. Indeed, we will

first formulate the constrained restoration

problem and obtain its solution in terms of a

general matrix Q.

21/11/15

Duong Anh Duc - Digital Image Processing

17

Constrained least squares

filtering (restoration)

Later different choices of matrix Q will be

considered, each giving rise to a different

restoration filter.

Suppose Q is any matrix (of appropriate

dimension). In constrained image restoration, we

choose ^f to minimize ||Qˆf||2, subject to the

constraint,

||g-Hˆf||2= ||n||2.

(Recall the degradation equation

g=Hˆf +n g-Hˆf = n.)

21/11/15

Duong Anh Duc - Digital Image Processing

18

Constrained least squares

filtering (restoration)

Introduction of matrix Q allows considerable

flexibility in the design of appropriate

restoration filters (we will discuss specific

choices of Q later). So our problem is

formulated as follows:

min ||Qˆf||2

subject to ||g-Hˆf||2= ||n||2 or ||g-Hˆf||2- ||n||2=0

21/11/15

Duong Anh Duc - Digital Image Processing

19

A brief review of matrix

differentiation

Suppose

x

x1

x2

and f(x1,x2) is a function of two variables. Then

f x1 , x2

x

21/11/15

f x1 , x2

x1

f x1 , x2

x2

Duong Anh Duc - Digital Image Processing

20

A brief review of matrix

differentiation

If

f(x1,x2) = ||Ax-b||2 = (Axb)T (Axb)

for some matrix A and some vector b , then

f x1 , x2

x

T

2A Ax b

where superscript T denotes matrix

transpose.

21/11/15

Duong Anh Duc - Digital Image Processing

21

A brief review of matrix

differentiation

Recall from calculus that such a constrained

minimization problem can be solved by means of

Lagrange multipliers. We need to minimize the

augmented objective function J(ˆf):

J(ˆf) = ||Qˆf||2+ (||g-Hˆf||2- ||n||2),

where is a Lagrange multiplier.

We set the derivative of J(ˆf) with respect to ˆf to

zero.

J(ˆf) = 2QT Qˆf-2 HT(g-Hˆf) = 0

(QT Q + HTH)ˆf = HTg

21/11/15

Duong Anh Duc - Digital Image Processing

22

A brief review of matrix

differentiation

Therefore,

ˆf = (QT Q + HTH)-1 HTg

= (QT Q/ + HTH)-1HTg

= ( QT Q + HTH)-1HTg

where = 1/ is chosen to satisfy the

constraint ||g-Hˆf||2= ||n||2.

We will now use the above formulation to

derive a number of restoration filters.

21/11/15

Duong Anh Duc - Digital Image Processing

23

Pseudo-inverse Filtering

The pseudo-inverse filter tries to avoid the

pitfalls of applying an inverse filter in the

presence of noise.

Consider the constrained restoration

solution,

^f = ( QT Q + HTH)-1HTg

with Q = I. This gives,

^f = ( I + HTH)-1HTg

21/11/15

Duong Anh Duc - Digital Image Processing

24

Pseudo-inverse Filtering

It can be implemented in the Fourier domain by

the following equation:

^F(u,v) = R(u,v)G(u,v), where

R u, v

*

H u, v

H u, v

2

1

H u, v

H u, v

H u, v

2

2

The parameter is a constant to be chosen.

Note that = 0 gives us back the inverse filter. For

> 0, the denominator of R(u, v) is strictly positive

and the pseudo-inverse filter is well defined.

21/11/15

Duong Anh Duc - Digital Image Processing

25

## Digital Image Processing Part II pdf

## Cấu trúc dữ liệu 1 giáo trình trần hạnh nhi, dương anh đức, đh khoa học tự nhiên tp.hcm, 1996

## gonzalez, woods - digital image processing. instructors manual. 2nd

## wilhelm burger, mark j. burge - principles of digital image processing. fundamental techniques

## william k. pratt - digital image processing 2007

## cristobal, schelkens, thienpont - optical and digital image processing

## focal press digital image processing with application to digital cinema first edition dec 2005 ebook - ddu

## topological algorithms for digital image processing

## william k. pratt - digital image processing 3rd

## dahlhaus, kurths, maass, timmer - mathematical methods in signal processing and digital image analysis

Tài liệu liên quan