DTN : Image Translation with GAN (3)

DTN : Image Translation with GAN (3)

2. Unsupervised Cross-Domain Image Generation (DTN)

published to ICLR2017 by Yaniv Taigman, Adam Polyak, Lior Wolf

Learn $ G: S \rightarrow T $ of two related domains, $ S $ and $ T $ without labels! (labels of images are usually expensive)


Baseline model

inline 80%
$ D $ : discriminator, $ G $ : generator,
$ f $ : context encoder. outputs feature. (128-dim)

R_{GAN} = \max_D \mathbb{E}_{x\sim\mathcal{D}_S} \log[1-D(G(x))] + \mathbb{E}_{x\sim\mathcal{D}_T} \log[D(x)]

R_{CONST} = \mathbb{E}_{x\sim\mathcal{D}_S} d(f(x),f(G(x)))

$f$-constancy : Does $x, G(x)$ have similar context?
$ d $ : distance metric. ex) MSE
$ f $ : "Pretrained" context encoder. Parameter fixed.
$f$ can be pretrained with classification task on $S$

Minimize two Risks : $ R_{GAN}$ and $ R_{CONST} $

Experimentally, Baseline model didn't produce desirable results.
Thus, similar but more elaborate architecture proposed

Proposed "Domain Transfer Network (DTN)"


First, $ f $ : the context encoder now encode as $f(x)$ then $g$ will generate from it : $ G = g(f(x)) $

$g$ focuses to generate from given context $f(x)$

Second, for $x \in \mathbf{t}$, $x$ is also encoded by $f$ and applied $g$

"Pretrained $f$ on $S$" would not be good as much as on $T$. But enough for context encoding purpose
$ L_{TID}$ : $G(x)$ should be similar to $x$
Also $D$ takes $G(x)$ and performs ternary (3-class) classification. (one real, two fakes)


Discriminator loss : $L_D$

L_D = -\mathbb{E}_{x \in \mathbf{s}} \log D_1 (G(x)) - \mathbb{E}_{x \in \mathbf{t}} \log D_2 (G(x)) - \mathbb{E}_{x \in \mathbf{t}} \log D_3 (x)

$D_i(x): Probability$
$D_1(x)$ : generated from $S$? / $D_2(x)$ : generated from $T$? / $D_3(x)$ : sample from $T$?

Generator : Adversarial Loss $L_{GANG}$

L_{GANG} = - \mathbb{E}_{x \in \mathbf{s}} \log D_3 (G(x)) - \mathbb{E}_{x \in \mathbf{t}} \log D_3(G(x))

Fool $D$ to classify as sample from $T$

Generator : $L_{CONST}$ and Identity preserving $ L_{TID}$

L_{CONST} = \sum_{x \in \mathbf{s}} d(f(x),f(g(f(x)))
, in feature level

L_{TID} = \sum_{x \in \mathbf{t}} d_2(x,G(x))
, in pixel level

$d, d_2$ used as MSE in this work

L_{G} = L_{GANG} + \alpha L_{CONST}+ \beta L_{TID} + \gamma L_{TV}

$L_{TV}$ is for output smoothing.

$L_G$ minimized over $g$
$L_D$ minimized over $D$


  1. Street View House Numbers (SVHN) $\rightarrow$ MNIST
  2. Face $\rightarrow$ Emoji
    Both cases, $S$ and $T$ domains differ considerably
    SVHN $\rightarrow$ MNIST
    Pretrain $f$ on $SVHN_{f_TRAIN}$
    Learn $G: SVHN_{DTN_TRAIN} \rightarrow MNIST_{TEST}$
    Train a MNIST classifier on $MNIST_{TRAIN}$. will be used as evaluation purpose later
    Domain transfer on $SVHN_{TEST}$ : $G(SVHN_{TEST})$



  • 4 convs (each filters 64,128,256,128) / max pooling / ReLU
  • input $32 \times 32$ RGB / output 128-dim vector.
  • $f$ do not need to be very powerful classifier.
  • achieves 4.95% error on SVHN test set
  • Weaker in $T$ : 23.92% error on MNIST.
  • Learn analogy of unlabeled examples


  • Inspired by DCGAN
  • SVHN-trained $f$'s 128D representation $\rightarrow 32\times32$
  • four blocks of deconv, BN, ReLU. TanH at final.
  • $$ L_{G} = L_{GANG} + \alpha L_{CONST}+ \beta L_{TID} + \gamma L_{TV} $$

$\alpha=\beta=15, \gamma=0$

Evaluate DTN

Train classifier on $MNIST_{TRAIN}$.
Architecture same as $f$
MNIST performance 99.4% test set.
Evaluate by testing MNIST classifier on $ G(\mathbf{s}_{TEST}) = { G(x)|x \in \mathbf{s}_{TEST} } $ using $Y$ : $\mathbf{s}_{TEST}$ label.

right fit
left fit

Experiments: Unseen Digits

Study the ability of DTN to overcome omission of a class in samples.
For example, class '3' Ablation applied on

  1. training DTN, domain $S$
  2. training DTN, domain $T$
  3. training $f$.

But '3' exists in testing DTN! Compare results.
right fit
a inlineb inlinec inlined inlinee inlinef inline
(a) The input images. (b) Results of our DTN. (c) 3 was not in SVNH. (d) 3 was not in MNIST. (e) 3 was not shown in both SVHN and MNIST. (f) The digit 3 was not shown in SVHN, MNIST and during the training of f.


Domain Adaptation

$S$ labeled, $T$ unlabeled, want to train classifier of $T$
Train k-NN classifier
right fit

Face $\rightarrow$Emoji

  • face from Facescrub/CelebA
  • emoji gained from bitmoji.com, not publicized
    preprocess on emoji with heuristics. Align face.
  • $f$ from DeepFace pretrained network. (Taigman et al. 2014) the author's previous work
  • $f(x)$ is 256-dim
  • $g$ outputs $64 \times 64$
  • SR (Dong et al. 2015) to upscale final output.


choose $\alpha=100, \beta=1, \gamma=0.05$ via validation
right fit

Original style transfer can't solve it


DTN also can style transfer.
DTN is more general than Styler Transfer method.


right fit

  • $f$ usually can be trained in one domain, thus asymmetric.
  • Handle two domains differently.
    • $T \rightarrow S$ is bad.
  • Bounded by $f$. Needs pre-trained context encoder.
  • any better way to learn context without pretraining?
  • Any more $S \rightarrow T$ tasks?


  • Demonstrate Domain Transfer, as an unsupervised method.
  • Can be generalized to various $S \rightarrow T$ problems.
  • $f$-constancy to maintain context of domain $S$ & $T$
  • Simple domain adaptation and good performance
  • inspiring work to future domain adaptation research

More open reviews at OpenReview.net