> ## Content Index
> Fetch the complete content index at: https://sudormrf.run/llms.txt
> Use this file to discover other available public pages before exploring further.

# DTN : Image Translation with GAN (3)
- URL: https://sudormrf.run/2018/04/02/image-translation-with-gan-3/
- Published: 2018-04-02T02:50:24.000Z
- Updated: 2018-04-04T14:25:21.000Z
- Author: Junho Cho
- Tags: Computer Vision, Deep Learning, #Import 2026-10-24 03:36

# 2\. Unsupervised Cross-Domain Image Generation (DTN)

published to **ICLR2017** by Yaniv Taigman, Adam Polyak, Lior Wolf

Learn $ G: S \\rightarrow T $ of two related domains, $ S $ and $ T $ without labels! (labels of images are usually expensive)

![inline](https://tmmsexy.s3.amazonaws.com/2017-03-29-155516.jpg)  
![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-100007.jpg)![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-121602.jpg)

## Baseline model

![inline 80%](https://tmmsexy.s3.amazonaws.com/2017-03-29-161854.jpg)  
$ D $ : discriminator, $ G $ : generator,  
$ f $ : context encoder. outputs feature. (128-dim)

\\begin{equation}  
R\_{GAN} = \\max\_D \\mathbb{E}\_{x\\sim\\mathcal{D}\_S} \\log\[1-D(G(x))\] + \\mathbb{E}\_{x\\sim\\mathcal{D}\_T} \\log\[D(x)\]  
\\end{equation}

\\begin{equation}  
R\_{CONST} = \\mathbb{E}\_{x\\sim\\mathcal{D}\_S} d(f(x),f(G(x)))  
\\end{equation}

$f$-constancy : Does $x, G(x)$ have similar context?  
$ d $ : distance metric. ex) MSE  
$ f $ : "Pretrained" context encoder. *Parameter fixed*.  
$f$ can be pretrained with classification task on $S$

Minimize two Risks : $ R\_{GAN}$ and $ R\_{CONST} $

Experimentally, Baseline model didn't produce desirable results.  
Thus, similar but more elaborate architecture proposed

## Proposed "Domain Transfer Network (DTN)"

![inline](https://tmmsexy.s3.amazonaws.com/2017-03-29-142809.jpg)

First, **$ f $ : the context encoder** now encode as $f(x)$ then $g$ will generate from it : $ G = g(f(x)) $

$g$ focuses to generate from given context $f(x)$

Second, for $x \\in \\mathbf{t}$, $x$ is also encoded by $f$ and applied $g$

"Pretrained $f$ on $S$" would not be good as much as on $T$. But enough for context encoding purpose  
$ L\_{TID}$ : $G(x)$ should be similar to $x$  
Also $D$ takes $G(x)$ and performs ternary (3-class) classification. (one real, two fakes)

## Losses

### Discriminator loss : $L\_D$

\\begin{equation}  
L\_D = -\\mathbb{E}\_{x \\in \\mathbf{s}} \\log D\_1 (G(x)) - \\mathbb{E}\_{x \\in \\mathbf{t}} \\log D\_2 (G(x)) - \\mathbb{E}\_{x \\in \\mathbf{t}} \\log D\_3 (x)  
\\end{equation}

$D\_i(x): Probability$  
$D\_1(x)$ : generated from $S$? / $D\_2(x)$ : generated from $T$? / $D\_3(x)$ : sample from $T$?

### Generator : Adversarial Loss $L\_{GANG}$

\\begin{equation}  
L\_{GANG} = - \\mathbb{E}\_{x \\in \\mathbf{s}} \\log D\_3 (G(x)) - \\mathbb{E}\_{x \\in \\mathbf{t}} \\log D\_3(G(x))  
\\end{equation}

Fool $D$ to classify as sample from $T$

### Generator : $L\_{CONST}$ and Identity preserving $ L\_{TID}$

\\begin{equation}  
L\_{CONST} = \\sum\_{x \\in \\mathbf{s}} d(f(x),f(g(f(x)))  
\\end{equation}  
, in feature level

\\begin{equation}  
L\_{TID} = \\sum\_{x \\in \\mathbf{t}} d\_2(x,G(x))  
\\end{equation}  
, in pixel level

$d, d\_2$ used as MSE in this work

\\begin{equation}  
L\_{G} = L\_{GANG} + \\alpha L\_{CONST}+ \\beta L\_{TID} + \\gamma L\_{TV}  
\\end{equation}

$L\_{TV}$ is for output smoothing.

$L\_G$ minimized over $g$  
$L\_D$ minimized over $D$

## Experiments

1. Street View House Numbers (SVHN) $\\rightarrow$ MNIST
2. Face $\\rightarrow$ Emoji  
Both cases, $S$ and $T$ domains differ considerably  
SVHN $\\rightarrow$ MNIST  
Pretrain $f$ on $SVHN\_{f\_TRAIN}$  
Learn $G: SVHN\_{DTN\_TRAIN} \\rightarrow MNIST\_{TEST}$  
Train a MNIST classifier on $MNIST\_{TRAIN}$. will be used as evaluation purpose later  
Domain transfer on $SVHN\_{TEST}$ : $G(SVHN\_{TEST})$

![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-114629.jpg)

### $f$

- 4 convs (each filters 64,128,256,128) / max pooling / ReLU
- input $32 \\times 32$ RGB / output 128-dim vector.
- $f$ do not need to be very powerful classifier.
- achieves 4.95% error on SVHN test set
- Weaker in $T$ : 23.92% error on MNIST.
- Learn analogy of unlabeled examples

### $g$

- Inspired by DCGAN
- SVHN-trained $f$'s 128D representation $\\rightarrow 32\\times32$
- four blocks of deconv, BN, ReLU. TanH at final.
- $$ L\_{G} = L\_{GANG} + \\alpha L\_{CONST}+ \\beta L\_{TID} + \\gamma L\_{TV} $$

$\\alpha=\\beta=15, \\gamma=0$  
![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-100007.jpg)![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-100056.jpg)

### Evaluate DTN

Train classifier on $MNIST\_{TRAIN}$.  
Architecture same as $f$  
MNIST performance 99.4% test set.  
Evaluate by testing MNIST classifier on $ G(\\mathbf{s}\_{TEST}) = { G(x)|x \\in \\mathbf{s}\_{TEST} } $ using $Y$ : $\\mathbf{s}\_{TEST}$ label.

![right fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-114629.jpg)  
![left fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-100214.jpg)

## Experiments: Unseen Digits

Study the ability of DTN to overcome omission of a class in samples.  
For example, class '3' Ablation applied on

1. training DTN, domain $S$
2. training DTN, domain $T$
3. training $f$.

But '3' exists in testing DTN! Compare results.  
![right fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-115440.jpg)  
![a inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111009.jpg)![b inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111020.jpg)![c inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111048.jpg)![d inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111102.jpg)![e inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111120.jpg)![f inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-111137.jpg)  
(a) The input images. (b) Results of our DTN. (c) 3 was not in SVNH. (d) 3 was not in MNIST. (e) 3 was not shown in both SVHN and MNIST. (f) The digit 3 was not shown in SVHN, MNIST and during the training of f.

![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-115645.jpg)

## Domain Adaptation

$S$ labeled, $T$ unlabeled, want to train classifier of $T$  
Train k-NN classifier  
![inline](https://tmmsexy.s3.amazonaws.com/2017-03-31-052449.jpg)  
![right fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-104539.jpg)

### Face $\\rightarrow$Emoji

- face from [Facescrub](http://vintage.winklerbros.net/facescrub.html?ref=sudormrf.run)/[CelebA](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html?ref=sudormrf.run)
- emoji gained from [bitmoji.com](https://www.bitmoji.com/?ref=sudormrf.run), not publicized  
preprocess on emoji with heuristics. Align face.
- $f$ from DeepFace pretrained network. (Taigman et al. 2014) the author's previous work
- $f(x)$ is 256-dim
- $g$ outputs $64 \\times 64$
- SR (Dong et al. 2015) to upscale final output.

### Results

choose $\\alpha=100, \\beta=1, \\gamma=0.05$ via validation  
![inline](https://tmmsexy.s3.amazonaws.com/2017-03-31-010439.jpg)  
![right fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-121602.jpg)

Original style transfer can't solve it

![inline](https://tmmsexy.s3.amazonaws.com/2017-03-30-122508.jpg)

DTN also can style transfer.  
DTN is more general than Styler Transfer method.

## Limitations

![right fit](https://tmmsexy.s3.amazonaws.com/2017-03-30-123135.jpg)

- $f$ usually can be trained in one domain, thus asymmetric.
- Handle two domains differently.  
  - $T \\rightarrow S$ is bad.
- Bounded by $f$. Needs pre-trained context encoder.
- any better way to learn context without pretraining?
- Any more $S \\rightarrow T$ tasks?

## Conclusion

- Demonstrate Domain Transfer, as an unsupervised method.
- Can be generalized to various $S \\rightarrow T$ problems.
- $f$-constancy to maintain context of domain $S$ & $T$
- Simple domain adaptation and good performance
- inspiring work to future domain adaptation research

More open reviews at [OpenReview.net](https://openreview.net/forum?id=Sk2Im59ex¬eId=Sk2Im59ex&ref=sudormrf.run)