c# itextsharp pdf to image: Open password protected pdf application SDK tool html wpf web page online enterface06

MultimodalSpeakerConversion

—hismaster’svoice...andface—

T.Dutoit

∗

,A.Holzapfel

†

,M.Jottrand

∗

,F.Marqu´es

‡

,A.Moinet

∗

F.Oﬂi

,J.P´erez

‡

andY,Stylianou

†

∗

FacultePolytechniquedeMons-BELGIUM

†

UniversityofCrete-GREECE

‡

UniversitatPolit`ecnicade

Catalunya-SPAIN

KocUniversity-TURKEY

Abstract—The goal of this project is to convert a given

speaker’s speech (the Source speaker) into another identiﬁed

voice (the Target speaker) as well as analysing the face animation

of the source to animate a 3D avatar imitating the source

facial movements. We assume we have at our disposal a large

amount of speech samples from the source and target voices

with a reasonable amount of parallel data. Speech and video are

processed separately and recombined at the end.

Voice conversion is obtained in two steps: a voice mapping step

followed by a speech synthesis step. In the speech synthesis step,

we speciﬁcally propose to select speech frames directly from the

large target speech corpus, in a way that recall the unit-selection

principle used in state-of-the-art text-to-speech systems.

The output of this four weeks work can be summarized as: a

tailored source database, a set of open-source MATLAB and C

ﬁles and ﬁnally audio and video ﬁles obtained by our conversion

method. Experimental results show that we cannot aim to reach

the target with our LPC synthesis method; further work is

required to enhance the quality of the speech.

Index Terms—voice conversion, speech-to-speech conversion,

speaker mapping, face tracking, cloning, morphing, avatar con-

trol.

I. I

NTRODUCTION

HIS project aims at converting a given speaker speech

and facial movements into those of another (identiﬁed)

speaker. More precisely, it focuses on controlling a 3D talking

face (the avatar of the target speaker) using the most natural

interfaces available : the speech and facial movements of a

human speaker (the source speaker). It also assumes that the

target will generally be different from the source, so that the

project goes much further than the design of a state-of-the-art

avatar controlled by the original speaker (i.e., the same as the

one whose face was used to create the avatar). As a matter of

fact, two additional problems are encountered here:

•

That of voice conversion, in order to make the target

talking face speak with the target’s voice, producing the

source’s words.

•

That of facial movement conversion, in order to adapt

the movement excursions of the source face to match the

movement excursions of the target face.

The multimodal conversion problem we consider here,

however, is limited to signal-level modiﬁcation, as opposed

to semantic conversion. The latter would enable, for instance,

ﬁltering out some (most often paralinguistic) communication

This report, as well as the source code for the software developed

during the project, is available online from the eNTERFACE’06 web site:

http://www.enterface.net/enterface06.

acts (be them speech and/or facial movements, such as tics)

of the source which the target never produces, or conversely

adding target movements not present in the source but usually

in the target.

Moreover, we constraint the conversion process to maintain

some large-sense synchronicity between source and target: we

do not aim at adapting speech rate at the phoneme level,

but rather simplifying it to a possible overall speech rate

adaptation. Similarly, we do not consider a possible syllable-

level F0 conversion from source to target, but rather aim at a

possible overall F0 adaptation ratio.

It will be assumed that a large amount of studio-quality

speech data is available from the source and from the target.

This is not a usual assumptionfor systems which tryto put new

words in the (virtual) mouth of most VIP characters (whom

cannot be easily forced to attend a formal recording session

in a studio). The assumption, however, remains realistic in the

case of a source speaker drivinga famous target speaker whose

voice has been recorded in large amounts but who is simply

no longer (or not always) available. It is also assumed that this

speech data is available in the form of parallel speech corpora

(i.e., recordings of the same sentences by both the source and

the target).

Atypical application of this project is therefore that of a

human actor controlling the speech and facial movements of

a 3D character whose voice is well-known to the audience.

Another possible use is for psychologists talking to children

through an avatar whose voice should be kept unchanged

among sessions, even though the psychologist may change.

If we reduce this project to its speech component, a typical

application is that of a tool for producing natural sounding

voice prompts with the voice of a voice talent, based on natural

speech (prosody and spectral features) produced by a human

speaker.

Last but not least, a side constraint of this work is that we

aim at using and producing open-source code, as required by

the eNTERFACE workshop organization.

Figure 1 shows the necessary steps involved in our project:

speech/face analysis, voice/facial movements mapping, and

speech/face synthesis. This report is therefore organized as

follows. Section II browses the state-of-the-art in speech

analysis, mapping, and synthesis for voice conversion, and

examines the approach we have followed in this project.

Section III examines facial movement analysis, mapping and

synthesis for avatar face animation, and gives details on the

algorithms we have used. This is followed in section IV by

Open password protected pdf - C# PDF Password Library: add, remove, edit PDF file password in C#.net, ASP.NET, MVC, WinForms, WPF

Help to Improve the Security of Your PDF Document by Setting Password

convert password protected pdf to normal pdf online; change password on pdf document

Open password protected pdf - VB.NET PDF Password Library: add, remove, edit PDF file password in vb.net, ASP.NET, MVC, WinForms, WPF

Help to Improve the Security of Your PDF Document by Setting Password

pdf protected mode; add password to pdf preview

Fig. 1. An example of 3D cartoon avatar control using source speech and facial movements, and mapping them to target speech and face.

the experiments we did, using a specially designed database,

and by some informal assesment of the quality we reached

at the end of the 4-weeks project. The paper is concluded in

Section V by perspectives for further developments.

II. S

PEECH ANALYSIS

MAPPING

AND SYNTHESIS FOR

VOICE CONVERSION

We understand the Voice Conversion (VC) process as a

modiﬁcation of a speaker voice (source speaker) so that it

resembles that of another given speaker (target speaker). Our

goal is to obtain a transformation that makes the source speech

sound as if it were spoken by the target speaker.

In this work, we have approached the speech conversion

task from x (source speaker) to y (target speaker) in two

independent blocks:

•

mapping from x to y



(a ﬁrst approximate of y) using the

(reduced) parallel corpus, and

•

speech-to-speech (S2S) synthesis from y



to y



(a second

–and more accurate– approximate of y), using the (full)

target corpus.

The ﬁrst block involves aligning the data on a frame by

frame basis (section II-A) and building a mapping function

(either using Gaussian Mixture Models or Conditional Vector

Quantization, as will be seen in section II-B). In the second

block, for the successive frames of y, we select new frames

from a large database of y voice, in such a way that we

guarantee both maximum similarity to the input frames, and

maximum continuity (the details are given in section II-D.3).

This can be seen as a smoothing step, made necessary by the

fact that the reduced size of the parallel corpus resulted in

large, non-acceptable discontinuities in the synthetic speech.

It is worth mentioning that since a large amount of data

for the target is available for this particular application (as

it is also the case for the design of a state-of-the-art text-to-

speech (TTS) system), the challenge of this project is to be

able to produce more natural sounding speech than that of a

TTS system, trained with the same amount of data, and used to

synthesize the phonemes obtained by automatic phonetic seg-

mentation (speech recognition) of the input source waveform.

As a matter of fact, the input of the voice conversion system

is itself natural speech, which can hopefully be put to proﬁt

to deliver natural-sounding output speech. The intonation of

the source speech, for instance, can readily be used (possibly

modiﬁed) to produce the intonation of the target speech, and

thus obtain an improvement in synthesis quality over standard

TTS.

Furthermore, it is then possible to establish upper and lower

bounds to the synthesis quality we can obtain with our frame-

basedvoice conversion system. The lower bound would be that

obtained with a TTS backend. In this case, we would directly

use the prosody of the source waveform and the phonemes,

obtained with automatic speech recognition. The TTS would

then only need to perform unit selection and concatenation

based on this requirements. This way we expect to obtain

a higher similarity and naturality than when using only the

source (recognized) text as input. The upper bound would of

course be the natural target speech.

A. Data alignment

Although the corpus used for the voice mapping part of

+this project (see Section IV) consists of parallel utterances,

some timing differences are unavoidable due to different

speaker characteristics (pause durations, speech rate, etc.).

Since the training of the voice mapping block of ﬁg. 1 requires

parallel data vectors, the utterances of the source and target

speakers have been aligned using a dynamic time warping

(DTW) procedure. For this project, we used the DTW algo-

rithm implemented by Dan Ellis1 and released under GPL. The

local match measure is computed as the cosine distance (angle

of the vectors) between the Short-Time Fourier Transform

(STFT) magnitudes. The left part of ﬁg. 2 represents the

local match scores matrix, with darker zones indicating high

similarity values (ideally we would have a dark stripe down

the leading diagonal).In the right part of the ﬁgure we can see

the minimum-cost-to-this-point matrix (lighter color indicates

lower cost). In both subﬁgures, the red line shows the lowest-

cost path obtained by the DTW algorithm.

The DTW algorithm introduces some unavoidable errors

due to the coexistence of intrinsic spectral differences between

the two speakers. An iterative procedure can be applied

to improve the alignment, by reapplying the DTW method

between the converted and target envelopes [1]. After each

iteration, a new mapping function can be estimated between

the newly aligned original source and target data. In this

project, convergence was reached after three iterations.

B. Voice mapping

When converting the voice of the source to the voice of a

the target speaker we assume that these two voices are deﬁned

by their spectral spaces X and Y respectively. Our problem

1http://labrosa.ee.columbia.edu/matlab/dtw/

Online Remove password from protected PDF file

Online Remove Password from Protected PDF file. Download Free Trial. Remove password from protected PDF file. Find your password-protected PDF and upload it.

add password to pdf online; add password to pdf file with reader

VB.NET PDF Convert to Jpeg SDK: Convert PDF to JPEG images in vb.

NET PDF Document Viewer, C#.NET PDF Windows Viewer, C#.NET convert image to PDF, C#.NET convert open office to Able to convert password protected PDF document.

break password pdf; add password to pdf file

Fig. 2. Left: Graphicalrepresentation of thelocalmatch scores matrix(darker

colors represent a closer match). Right: Minimum cost matrix (lighter colors

indicate alowercost).The red line shows the optimal alignmentpath as found

by the DTW algorithm.

in voice conversion is two-fold: at ﬁrst we have to ﬁnd a way

to model these spaces and then we have to ﬁnd a way to

map a previously unknown example from the source space to

the target space. In order to be able to ﬁnd such a mapping

we assume that there is aligned training data available. This

means that we have two sets of spectral vectors x

and y

that

describe spectral envelopes from source and target speakers

respectively. The two sets of vectors {x

,t = 1,...,N} and

,t = 1,...,N} have the same length N and are supposed

to describe sentences uttered in parallel by source and target.

What is desired is a function F() such that the transformed

envelope F(x

) best matches the target envelope y

, for all

envelopes in the learning set (t = 1,...,N).

In our project we tried two different approaches in order to

achieve the goal of conversion: Gaussian Mixture Models and

Conditional Vector Quantization.

a) Gaussian Mixture Models: The ﬁrst approach has

been described by Stylianou et al.[1] and is based on a de-

scription of the source space using Gaussian Mixture Models:

p(x) =

M

i=1

N(x;µ

,Σ

)

(1)

where M is the number of Gaussians, µ

,Σ

are the mean

vector and the covariance matrix of the i-th Gaussian com-

ponent, and α

are the weights used to combine the different

components. These M Gaussian components can be regarded

as classes within the spectral space of the source and a vector

can be classiﬁed to one of the classes using maximum

likelihood. The mapping to the target space is done by using

these parameters in the conversion function

F(x

M

i=1

P(C

)



+Γ

−1

−µ

)



(2)

where ν and Γ are related to the mean target and the

cross-covariance matrix of the source and target vectors. The

parameters of the conversion function are determined by

minimization of the total quadratic spectral distortion between

the converted envelopes and the target envelopes:

=

N

t=1

||y

−F(x

)||

(3)

For details on the minimization see [1].

b) Conditional Vector Quantization: The second method

for voice conversion applies a Conditional Vector Quantization

as presented in [2]. In contrast to the ﬁrst method we get hard

cluster boundaries by using a standard LBG clustering for the

source space giving us a codebook C

≡ {˙x

,i = 1...m}.

Then the mapping function ﬁnds for each of these clusters a

different codebook C

with k entries for each source space

cluster. The criterion function minimized in this case is the

approximation to the average distortion D given by

D≈

M

m=1



N

n=1

p(˙x

)

K

k=1

p(˙y

m,k

|˙x

)d(y

,˙y

m,k

)



(4)

The conditional probability p(˙x

)is the association prob-

ability relating the input vector x

with codevector ˙x

,while

the association probability p(˙y

m,k

|˙x

)relates the output

vector y

with the codevector ˙y

m,k

of the m-th subcodebook

of C

The mapping for a source feature vector x

is done by

choosing the nearest source space cluster using Euclidean

distance. This provides us with a subcodebook of C

with K

entries. For an utterance of length N we construct a lattice of

K× N elements and we ﬁnd a minimum weight path using

again Euclidean distance from frame to frame providing us

with a sequence of N vectors in Y.

C. Frame selection algorithm

Once the features of the frames of the original speaker have

been converted using either the GMM mapping or the CVQ-

based conversion, the converted features are used as inputs of

the unit-selection algorithm.

This algorithm is basically working like any TTS unit-

selection system. However, state-of-the-art TTS systems based

on unit-selection usually deal with diphones, phones or parts

of phones [3] while our algorithm uses smaller units : 32 ms

frames (with a constant shift of 8 ms between each frame).

For this part of the system, we use the complete target

speech database (as opposed to the mapping system, which

only used the aligned sub-part of it; see Section IV for details

on the databases). Among all the frames in the target database

(cmu

awb

arctic ’95), we select a sequence of frames

Y =

[¨y

(1)

... ¨y

(t)

... ¨y

(T)

]that best matches the sequence of frames

output by the mapping function:

Y = [˙y

(1)

... ˙y

(t)

... ˙y

(T)

This selection is made using the Viterbi algorithm [4], [5] to

minimize the global distance between

Y and

Y. This global

distance is a combination of target and concatenation distances

which are detailed in the next three subsections. This approach

is very similar to that developed in Suenderman et al. [6], with

the difference that in our approach the target sequence for the

frame selection algorithm is the mapped sequence

Y, while

Suenderman et al. use the input sequence X as target.”

C# PDF Convert to Jpeg SDK: Convert PDF to JPEG images in C#.net

Support for customizing image size. Password protected PDF document can be converted and changed. Open source codes can be added to C# class.

add copy protection pdf; add password to pdf reader

VB.NET PDF Convert to Word SDK: Convert PDF to Word library in vb.

Create editable Word file online without email. Supports transfer from password protected PDF. VB.NET class source code for .NET framework.

a pdf password online; pdf password security

1) Clustering, clusters and n-best selection: In order to

reduce the computation time, the database has been divided

into 256 clusters using the LBG method [7]. When we use

only the parallel recordings as a database, there are about 300

frames per cluster and each cluster has a centroid which is

a vector of the mean values of the features of the frames

inside the cluster. Therefore, for each set of features ˙y(t), the

algorithm ﬁrst selects the cluster with the closest centroid.

The closeness of the centroid is measured using a weighted

euclidean distance

closest centroid = argmin

c=1,...,C

N

i=1



˙y

(t)

−¨y

(c)



, (5)

where N is the dimension of the feature vectors, ˙y

(t)

is the

component of the feature vector produced by the mapping

function at time t, ¨y

(c)

is the i

component of the c

centroid

of the database and w

is the weighting factor associated to

that i

component.

Then, for eachframe (1,...,m

,...,M

)inthe chosen cluster

c, the weighted euclidean distance between the feature vectors

y˙

(t)

and ¨y

)

is computed. This distance will be used as the

target distance t

dist

by the Viterbi algorithm:

dist

(t,m

N

i=1



˙y

(t)

−¨y

)



(6)

Finally, if the n-best option is activated, only the NBEST

closest ¨y

)

to the y˙

(t)

are selected as candidates for Viterbi.

2) Concatenation distance: During the induction step, the

Viterbi algorithm has to compute the cost of all the transitions

from each feature vector ¨y

c(t)

)

,selected at time instant t, to

each vector ¨y

c(t+1)

)

, selected at time t + 1. Again, these

concatenation costs (c

dist

) are measured using a weighted

euclidean distance:

dist

c(t)

c(t+1)

N

i=1



¨y

c(t+1)

)

−¨y

c(t)

)



(7)

This step is the most time-consuming part of the Viterbi

algorithm and is the reason why clusters and n-best selections

are so important. They reduce the transition possibilities but

in return they dramatically increase the search speed without

actually nor notably damaging the ﬁnal output.

At this point, one important remark has to be done :

the concatenation distance should advantage the selection of

neighbour frames to reduce discontinuities during the synthesis

of speech. Therefore, before computing the distance between

two feature vectors, the process checks whether the corre-

sponding frames are consecutive in a wav ﬁle of the database.

In case they are, the distance is automatically set to zero.

3) Implementation: The very ﬁrst tests were made using

Matlab, however it early appeared that this was very time-

consuming. In order to reduce the duration of the Viterbi

algorithm, the whole program has been rewritten in C so that

it could be compiled in a mex-ﬁle. Mex-ﬁles are pre-compiled

libraries used by Matlab as any other Matlab script. The

difference with usual scripts is that the functions implemented

in these libraries are way faster (see Annex V-G in p. 11).

Fig. 3. Block diagram of the selection method using a projection of the

mapped source MFCCs vectors into the target MFCCs space.

D. Speech synthesis

As a result of the previous steps (mapping, possibly fol-

lowed by a frame-selection algorithm as described in II-C),

we obtained a sequence of converted (target) frames that best

represent the voice transformation of a given source utterance.

The next step is to generate the converted speech resulting

from this sequence. Although it is possible to regenerate the

speech from the MFCC parameters [8], the resulting quality

is not acceptable for the task, since the parametrization is

not a lossless procedure. In particular, the phase information

is completely lost and must be estimated to provide the

synthetic voice with a higher degree of naturalness. In this

subsection, we ﬁrst produce ¨y(n) using the source speaker

excitation as input of LPC ﬁlters modelling the target speech.

We then examine the production of ¨y(n) by overlap-adding

speech frames (i.e., samples) extracted directly from the target

database.

1) LP-based speech synthesis, without Viterbi: This method

is illustrated in ﬁg. 3. The speech signal is segmented into

overlapping frames and MFCCs are computed for each frame

(vectors X in the ﬁgure). The mapping function (continuous

probabilistic mapping function based on GMM and described

earlier) computes transformed MFCCs vectors (

Y). Each

Y is

then compared to all target database MFCC vectors in order

to get the closest match

Y . For each target MFCC vector,

we also have memorized the corresponding auto-regressive

coefﬁcients. Thus, for each frame of the source speaker, we

can get the corresponding autoregressive coefﬁcients for the

target speaker.

In parallel, for each source frame, we run a LPC analysis

to extract the LP residual of the source. Finally, to synthesize

converted speech, we use this residual as input excitation for

the LPC synthesis ﬁlter with the auto-regressive coefﬁcients

obtained.

We know that to keep the excitation of the source is not an

ideal solution since it still keeps information from the source.

To approach a bit more the target voice, a pitch modiﬁcation

can be done on the converted speech by analysing the pitch

of the source and of the target.

2) LP-based speech synthesis, with Viterbi: The voice

conversion system by LPC analysis and synthesis using the

Viterbi algorithm is detailed in ﬁg. 4. As in the previous

method, the mapping function transforms MFCCs from the

source speech signal (X) into

Y MFCC vectors. For each

frame of the processed sentence, the Viterbi algorithm (that has

been previously described) gives as output an MFCC vector

C# PDF Convert to Word SDK: Convert PDF to Word library in C#.net

Password protected PDF file can be printed to Word for mail merge. C# source code is available for copying and using in .NET Class.

change password on pdf; convert password protected pdf to excel online

.NET PDF SDK - Description of All PDF Processing Control Feastures

PDF Document Protect. PDF Password. Able to Open password protected PDF; Allow users to add password to PDF; Allow users to remove password from PDF;

break pdf password online; pdf password unlock

Fig. 4. Block diagram of the selection method using the Viterbi algorithm.

taken directly from the MFCC vectors of the target speech

database. It also provides the index of the sentence the frame

chosen comes from and the time stamp of the centre of the

frame. From this information, one extracts the frame from its

wave ﬁle and computes an LPC analysis to get the new AR

coefﬁcients that will be used for the synthesis.

In comparison with the previous method, one can expect an

improvement in the choice of the frames in the target database,

since the Viterbi algorithm takes the concatenation cost into

account (and not only the target cost).

3) Speech to Speech synthesis: Instead of trying to produce

the converted speech samples by LPC analysis-synthesis, as in

the previous subsection, it is also possible to deliver speech

by overlap-adding speech frames taken directly from the target

speech ﬁles.

Here we use the natural target speech frames associated to

each MFCC vector. The problem then reduces to combining

these frames in order to achieve the highest quality possible.

We use a simple OverLap-and-Add (OLA) procedure on

the sequence of frames. However, there may be important

problems associated to the discontinuities between the frames.

To partially overcome this drawback, we apply a correction on

the frame positions, based on the local maximization of the

correlation between the already generated synthetic speech and

each new frame, as used in WSOLA [9].

III. F

ACIAL EXPRESSION ANALYSIS

MAPPING

AND

SYNTHESIS FOR FACE ANIMATION

Facial expression analysis and synthesis techniques have

received increasing interest in recent years. Numerous new

applications can be found, for instance in the areas of low bit-

rate communication or in the ﬁlm industry for animating 3D

characters.

In this project, we are interested in head-and-shoulder video

sequences, which we use to control the lip, eye, eyebrow,

and head movements of a 3D talking head. First a 3D head

model of the target is created (basically any 3D head will do,

including a cartoon-like head). This step typically involves

the extraction of Facial Deﬁnition Parameters (FDPs) from

photographs of the target speaker. The system then analyzes

the source video sequences and estimates 3D motion and facial

expressions using the 3D head model information. The expres-

sions are represented by a set of facial animation parameters

(FAPs) which are part of the MPEG4 standard [10] (just as

FDPs). The target 3D head model is then deformed according

to the FAPs of the source, and new frames are synthesized

using computer graphics techniques. Good examples of such

Fig. 5. An ellipse is ﬁtted to skin blob and optical ﬂowvectors are calculated

in this region

video-based avatar control can be found in the work of

Eisert [11].

A. Face Analysis

Face analysis process mainly consists of three tasks: de-

tecting and tracking face region, extracting features for facial

gestures from the detected face region, and tracking these

features throughout the source video. In this framework,

automatic face detection and computation of 8 quadratic global

head motion parameters is done according to [12]. Figure 5

shows an ellipse ﬁtted to the detected face region and optical

vectors calculated between two consecutive frames that will

be used in the process of computing quadratic global head

motion parameters.

The global head motion parameters are used to

•

estimate the position of head in the next frame,

•

approximate the 3D movement of the head and

•

calculate the canonical displacement vectors of facial

feature points over the face region.

Once the face position is known, a set of useful feature

points for face components, which are lips, eyes, and eyebrows

in this scenario, are deﬁned and tracked. For this purpose,

Active Appearance Models (AAM) approach that was intro-

duced by Cootes, Edwards and Taylor, is used as a means of

modeling and tracking face components [13], [14], [15], [16].

AAM is, in essence, a combination of ideas from Eigen-

face Models [17], Point Distribution Models [18], and Ac-

tive Shape Models [19] and has its roots in model-based

approaches towards image interpretation named deformable

template models where a deformable template model can be

characterized as a model which, under an implicit or explicit

optimization criterion, deforms a shape to match a known

object in a given image [20]. Consequently, AAMs establish

a compact parametrization of shape and texture, as learned

from a representative training set. Thus, AAMs require model

training phase before they are used to process unseen images.

In this training phase, the AAM learns a linear model of the

C#: How to Add HTML5 Document Viewer Control to Your Web Page

new website or open your existing RasterEdge_Demo_Docs/pdf/demo_1.pdf" })); _tabDemoFiles.addCommand AppSettings.Get("resourceFolder"); protected void Page_Load

convert pdf password protected to word online; pdf file password

C# Image: How to Integrate Web Document and Image Viewer

RasterEdgeImagingDeveloperGuide8.0.pdf: from this user manual, you can to customize the way to open and store mode; public string fid; protected void Page_Load

adding password to pdf; annotate protected pdf

correlation between parameter displacements and the induced

residuals.

During search for a face in a new frame, residuals are

iteratively measured and used to correct the current parameters

with respect to the main model, leading to a better ﬁt. After

a few iterations, a good overall match is obtained [15] .

Applying this search method to each video frame using the

model obtained after training will constitute the tracking part

of the task. Figure 6 demonstrates a sequence of consecutive

frames as a result of the tracking task. The output of the

tracking process will be the set of 2D positions of key points

of the face components for each frame which will be input to

the facial movement mapping module.

B. Facial Movement Mapping

Since the previous module tracks pixel locations of the key

feature points of the face components and the next module will

synthesize a 3D head animation using an avatar model based

on MPEG-4 parameters, the mapping function will compute

MPEG-4 parameters from a set of 2D image locations.

MPEG-4 Facial Animation deﬁnes two sets of parameters:

the Facial Deﬁnition Parameter (FDP) set and the Facial Ani-

mation Parameter (FAP) set [21], [22]. These two sets provide

a common framework for animating a 3D face deformable

mesh model with the help of high-level and low-level facial

actions, closely related to facial muscle movements.

The ﬁrst set of parameters, FDPs, is used to deﬁne feature

points that are basic components in 3D face deformable

meshes, represented by a 3D set of vertices. The movements

of these vertices drive the deformations to be applied to the

model to animate the desired facial expressions. 84 feature

points on morphological places of the neutral head model are

deﬁned by MPEG-4 Facial Animation, as shown in ﬁgure 7.

The FDPs mainly serve for specifying how the face

mesh will deform according to the transformation parameters

(FAPs).

The second set of parameters, FAPs, on the other hand,

consists of a collection of animation parameters that modify

the positions of the previously deﬁned feature points and,

thus, can be used to create or change the model for each

of the desired facial expressions. There are 68 FAPs that

can be grouped in two categories: high-level and low-level

parameters. The number of high-level parameters is only two.

The ﬁrst one is visemes, which are the visual equivalents to

phonemes in speech. This parameter deﬁnes the mouth shapes

produced by the different possible phonemes. The second

high-level parameter corresponds to facial expressions and can

take 6values, one for each of the 6 archetypal emotions (anger,

disgust, fear, joy, sadness and surprise). The remaining 66 low-

level parameters are used for basic deformations applied to

speciﬁc morphological points on the face, like the top middle

outer-lip, the bottom right eyelid, etc... Because FAPs are

universal parameters and independent from the head model

geometry, MPEG-4 Facial Animation deﬁnes a set of 6 units,

the Facial Animation Parameter Units (FAPU), to normalize

the FAPs and make them independent of the overall face

geometry. Prior toanimation of a virtual face, the FAPs have to

Fig. 7. Feature points are used to deﬁne the shape of a face model and the

facial animation parameters are deﬁned by motion of some of these feature

points

be normalized by their corresponding FAPUs. As depicted in

ﬁgure below, the FAPUs are deﬁned as fractions of distances

between key facial features (i.e. eye-to-eye distance, angular

unit, etc.).

At this point, 8 quadratic global motion parameters are used

to separate the local movements of key feature points from the

global movement of head, since input value for each FAP is

independent from other FAPs.

One can think about this scenario: if head rotates around its

horizontal axis by 45 degrees, mouth will alsorotate withhead.

In the meantime, mouth just opens a little bit, meaning that

upper and lower lips move away from each other. And now,

if one looks at the big picture, middle point of upper lip will

seem to be moving neither just horizontally nor just vertically,

but along an inclined line in between. If the deformation value

for the middle point of the upper lip is calculated directly by

taking the difference of the 2D pixel locations between two

consecutive frames, the ﬁnal value will obviously be a wrong

input for the animation. Instead of this, having (x(t),y(t)) in

frame t, for frame t+1, one can estimate (ˆx(t+1), ˆy(t+ 1))

using global motion parameters (a

,...,a

)and (x(t),y(t)).

Fig. 6. Example image sequence that demonstrates the performance of tracking by AAMs

Fig. 8. Some of the feature points are used to deﬁne FAP units when the

face model is in its neutral state. Fractions of distances between the marked

key features are used to deﬁne FAPUs

Then, the difference between the measurement (x(t+1),y(t+

1)) and estimate (ˆx(t+ 1), ˆy(t+1)) can be considered as the

actual deformation of the model for the lips. Likewise, all the

values for the used FAPs can be calculated in a similar fashion.

Besides calculating the FAP values for different face com-

ponents, FAP values for the head movement itself have to

be computed as three rotation angles around the three axes

of the head. This is not an easy task since going from 2D

world to the 3D world with only one angle of view requires

sophisticated approximations, where there is no chance for

accurate results at all. There are model-based pose estimation

approaches using ellipsoidal models [23], or downhill simplex

optimization method and the combination of motion and

texture features [24], [25].

However, even though it is not as accurate as other methods,

it is also possible to simply approximate the rotation angles by

assumingthat the head center moves around over the surface of

asphere that is centered at the top of the spinal cord. When the

displacement of head is projected onto this sphere, the angles

of head rotation can be estimated approximately.

IV. E

XPERIMENTS AND

ESULTS

In order to design the application depicted in Fig. 1, we

needed to choose a source and target speaker, make sure we

could have access to a large amount of speech from the target

(for the speech synthesis module), of a reasonable amount

of aligned speech data for source and target (for the voice

mapping module), and of some test speech+video data from

the source (in order to test the complete system). We chose to

use the CMU-ARCTIC databases as target speech [26]. This

database was designed speciﬁcally for building unit-selection-

based TTS systems. It is composed of eight voices, speaking

1150 sentences each (the same sentences, chosen to provide

a phonetically balanced corpus). We thus decided to record

an audiovisual complementary database, for use as the source

data.

The eNTERFACE06

ARCTIC database we have created is

composed of 199 sentences, spoken by one male speaker, and

uniformly sampled from the CMU

ARCTIC database [26]. For

each sentence, an .txt, a .avi, and a .wav ﬁles are available. The

.avi ﬁle contains images with 320x240 pixels, 30 frames per

second, of the speaker pronouncing the sentence (Fs=44100

Hz). The .wav ﬁle contains the same sound recording as in

the .avi ﬁle, but resampled to 16 kHz.

The database was recorded using a standard mini-DV digital

video camera. The recording of the speech signal was realized

through the use of a high-quality microphone, specially con-

ceived for speech recordings. The microphone was positioned

roughly 30cm below the subject’s mouth, outside the camera

view.

The background consisted of a monochromatic dark green

panel that covered the entire area behind the subject, to allow

easier face detection and tracking. Natural lighting was used,

so that some slight illumination variation can be encountered

among the ﬁles (Fig. 2).

The recordings were made using the NannyRecord tool

provided by UPC Barcelona, which makes it possible for

the speaker to hear the sentence it has to pronounce twice

before recording it. The source speaker used for the recordings

were the “awb” speaker of CMU

ARCTIC. The eNTER-

FACE06

ARCTIC speaker was asked to keep the prosody

(timing, pitch movements) of the source, while using his own

acoustic realization of phonemes, and of course, his voice

(i.e., not trying to imitate the target voice). This particular

setting has made it possible for the eNTERFACE

ARCTIC

recordings to be pretty much aligned with the corresponding

CMU

ARCTIC recordings.

Following the standard approach, the parallel database was

further divided into three subsets:

•

development set, consisting of 188 sentences (of the total

198), used during the training phase of the different

algorithms (alignment, voice mapping, etc.),

•

validation set, used to avoid overﬁtting during the re-

ﬁnement of the model parameters (number of clusters,

GMMs, etc.),

•

evaluation set, used to obtain the results of the multi-

modal conversion.

It is worth mentioning that this last subset (evaluation) was not

present in any of the stages of the training. The results we have

obtained can therefore be expected to generalize smoothly to

any new data.

Mapping

Selection

Frame

WSOLA

(target: mfcc)

(source:mfcc)

a.GMM

b.CVQ

a.DP

b. By Example

y’[n]

{file,t}

Y’

Fig. 10. Block diagram show the different alternatives for the mapping

estimation and for the frame selection.

The features computed from the signals were either 13 or 20

MFCC’s and their ﬁrst and second derivatives; the frame rate

was chosen to be 8ms. For the computation of the derivatives

we relied on an implementation by Dan Ellis

and the other

signal processing steps were taken from the MA toolbox

by Elias Pampalk

. We also computed estimations of the

fundamental frequency by using functionalities provided by

the Praat software

.For fundamental frequency and for signal

energy we can also provide ﬁrst and second derivatives so that

for each frame the full set of features was: 20 MFCC’s, 20

∆MFCC’s, 20 ∆∆MFCC’s, f

,∆f

,∆∆f

,energy, ∆energy

and ∆∆energy resulting in a vector of 66 dimensions.

In ﬁgure 10, we can see the different alternatives we have

implemented for each of the blocks.

A. Alignment and voice mapping

After the ﬁrst alignment the Euclidean distance (L

norm)

between the source and the target MFCCs was: 1210.23. Then

the GMM-based mapping was applied and the same norm

was measured on the Transformed data and the Target Data:

604.35 (improvement: 50.08%). Then the Source data were

transformed and a new alignment was performed. Using the

new aligned data, a new mapping function was estimated

and again the Source data were transformed, and again the

norm between the transformed data and the Target data

was measured (397.63). The process was repeated and a new

measurement of the performance of the mapping function was

measured using the L

norm (378.18).

Without iterations, we achieve a 50% reduction of distance

between the target and the source data. This percentage

should be higher, and provides also an information on the

difference between the two speakers, and/or of the differences

in the recording conditions. After some iteration we arrive to

stable mapping function, with an improvement over the initial

distance between the source and the target data of: 68.76%.

Figure 11 shows the reduction of the distortion due to the

iteration of the algorithm.

B. On clustering parameters

The incremental alignment procedure used a Gaussian Mix-

ture Model of the source space with 128 components. This

parameter remained ﬁxed throughout the experiments. It had

inﬂuence on the construction of the aligned data as well as on

the mapping from source to target space, as this mapping is

http://labrosa.ee.columbia.edu/matlab/rastamat/deltas.m

http://www.ofai.at/˜elias.pampalk/ma/

4http://www.fon.hum.uva.nl/praat/

Fig. 9. Facial excerpts from the eNTERFACE06

ARCTIC database.

300

400

500

600

700

800

900

1000

1100

1200

1300

Distortion: l

norm on MFCC

Fig. 11. Euclidean distance between the transformed data and the target

data. Stability is reached after three iterations, with on overall improvement

of 86%.

TABLE I

EAN SIGNAL TO NOISE RATIO FOR VALIDATION FILE FRAMES MAPPED

TO SOURCE SPACE CLUSTERS

Number of Clusters

SNR

256

11.58 dB

512

12.04 dB

1024

12.46 dB

following equation 2 and so depends as well on the number

of mixture components.

For the computation of the CVQ-based mapping the source

space had to be clustered. Table I shows the average logarith-

mic signal to noise ratios for our ﬁve validation ﬁles. We can

see that increasing the number of clusters brings improvements

of about 1 dB. For the Y |X codebook size eight was chosen,

so that we had eight candidates for the

Y to choose from using

the Viterbi algorithm.

C. Face animation

For our speciﬁc scenario, each frame in the training set

has been manually labeled with 72 landmarks by using the

AAM-API [27] software (freely available for non-commercial

use such as research and education). The image in ﬁgure 12

shows an example of an annotated image from the training

set.

In order to obtain an accurate mapping from the 2D pixel

locations to the MPEG-4 parameters, the annotation of the

Fig. 12. Example of a training image labelled with 72 landmark points

images in the training set should closely match the MPEG-4

FDPs. In this particular task, the mapping process includes

calculation of 6 facial animation parameters units (FAPUs),

besides 44 low-level FAPs. Figure 13 shows the face model

used in this work, already available in the XFace project.

We have found it necessary to smooth the calculated values

for animation, since the measurements in the tracking process

are noisy and small scale differences in the parameters for

the simulation process may have large effects in the resulting

animation. Kalman Filter was used for this purpose with a

state model consisting of positions and velocities of all the

key feature points on the face.

2cm

As a result of using the techniques described above, several

video ﬁles have been produced from the evaluation subset of

the database. These results are available for tests in the archive

of our project on the eNTERFACE06 website. We created

parallel videos showing source speaker and target avatar side-

by-side in order to evaluate the face detection and tracking

algorithm. As it can be seen in the videos, both algorithms

are able to provide accurate results (although they are very

sensitive to the tuning of the parameters, and in some cases

they result in unreliable estimations).

Utterances using the three speech generation techniques

previously explained (sec. II-D) have been generated. For

comparison purposes, we have also generated an English

voice for the Festival Speech Synthesis System using the full

CMU ARCTIC database (except the evaluation subset). The

Fig. 13. The face model available in the XFace is used to create the desired

animation

phonetic segmentation was performed automatically, and thus

the resulting voice contains errors that would require manual

correction. Informal tests show that the LPC-based methods

(secs. II-D.1 and II-D.2) produce more natural, continuous

sound speech, than the speech to speech synthesis method

(sec.II-D.3). However, the speaker identity of the converted

speech is closer in the later case to the target speaker. The

highest identiﬁcation score was obtained by the Festival voice,

although the discontinuities due to the automatic segmentation

seriously affect the quality of the synthetic speech.

V. C

ONCLUSIONS

In this paper we described a multimodal speaker conversion

system, based on the simultaneous use of facial animation

detection, 3D talking face synthesis, and voice conversion. A

typical and important feature of the problem we have treated is

that a large amount of speech data is assumed to be available

for the target speaker.

During this work, two techniques have been studied for the

mapping of the source voice to the target voice: Gaussian

Mixture Models (GMMs) and Condition Vector Quantization

(CVQ). Several alternatives for the synthesis block have been

implemented, without achieving acceptable quality. Prelimi-

nary analysis of the results indicate that discontinuities in the

phase are causing major distortion in the synthetic speech.

Possible solutions and directions for future research are given

below.

A. Enhanced multimodal approach

Of the two conversion problems mentioned in the intro-

duction of this paper (voice and facial movements), we have

mostly focused on the ﬁrst, and assumed the second merely

reduced to a scaling of source to target movements. Clearly,

each human face has its own ways of producing speech, so

that the facial movement conversion step could still be widely

enhanced by submitting it to a more complex mapping stage.

One immediately notices that the approach followed here

is only weakly multimodal: it actually uses late multimodal

fusion. One of the obvious ideas that could be exploited in the

future is that of using speech for face movement detection,

and possibly face movements for helping voice mapping. A

further step could then be to study simultaneous face and

voice conversion. This would require having video data for

the target (which we did not have here). The speech-to-speech

component of our project could then be made multimodal in

essence, by using facial movements in the deﬁnition of target

and concatenation costs, for instance.

B. Weighted Euclidean distance

Until now, we have not actually used the weighted Euclidean

distance, instead we simply used the basic Euclidean distance.

However methods exist that allow the computation of optimal

weights for such a distance. A ﬁrst future improvement of our

results could be to apply one of those methods.

Another way to improve this part of the system could be

to compute the target and concatenation cost with other mea-

surements such as Mahalanobis, Kullback-Leibler or Itakura

distance.

C. F

mapping and continuity

Presently, the target speech is synthesized using the residual

excitation from the source speaker. Therefore the utterance has

the same prosody than the source. This is a major drawback

because we think this is the main reason why the ﬁnal output

sounds as a third speaker situated between the source and the

target.

The next step in the development of an efﬁcient open

source voice conversion system should be to create a mapping

function between the two speaker’s prosodies.

D. Phase continuity

In order to reduce the inﬂuence of the source speaker’s

voice in the ﬁnal result, we would like not to use his residual

excitation anymore. Indeed, this residual still contains a lot of

information about him.

The OLA solution proposed in section II-D.3 can be a solu-

tion. However, this will introduce a lot of phase discontinuity

(aside from the energy and pitch discontinuities which can be

handled more easily). We have found no elegant solution to

this problem, which requires further study

E. Pitch synchronous processing

Adifferent approach could be to use a PSOLA algorithm to

achieve the resynthesis. If PSOLA is used in the synthesis step,

then the absolute value of the pitch is of lesser importance as

atarget for the Viterbi alignment, however its delta and delta-

delta are still important. Indeed PSOLA can change the overall

pitch easily, but delivers lower quality speech when the shape

of the pitch curve is modiﬁed.

Documents you may be interested

c# itextsharp pdf to image : Open password protected pdf application SDK tool html wpf web page online enterface06_tdmjam0-part1067