> ## Content Index
> Fetch the complete content index at: https://sudormrf.run/llms.txt
> Use this file to discover other available public pages before exploring further.

# Normalization layer
- URL: https://sudormrf.run/2018/04/12/normalization-layer/
- Published: 2018-04-12T01:58:46.000Z
- Updated: 2018-04-12T04:37:31.000Z
- Author: Junho Cho
- Tags: Deep Learning, #Import 2026-10-24 03:36

What is Batch Normalization?  
Instance Normalization?  
Conditional Batch Normalization?  
Conditional Instance Normalization?

**Batch Normalization** is first introduced by Sergey Ioffe, Christian Szegedy. **It increased image classification performance significantly.**

- [arxiv](https://arxiv.org/abs/1502.03167?ref=sudormrf.run)  
![Pasted image](https://dynalist.io/u/bDoKdkIJlMClpWgljrA2Vve_)

Interested in "Conditional Batch Normalization (CBN)", here's wrap up of `normalization` layers.

refer to "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization"  
Well compared in this paper.

- [Torch](https://github.com/xunhuang1995/AdaIN-style?ref=sudormrf.run)
- [pytorch](https://github.com/naoto0804/pytorch-AdaIN/blob/master/function.py?ref=sudormrf.run)
- [pytorch](https://github.com/leehomyc/Style-Transfer-Adaptive-Instance-Normalization?ref=sudormrf.run)
- [arxiv](https://arxiv.org/pdf/1703.06868.pdf?ref=sudormrf.run)

## Batch Normalization (BN)

$$\\text{BN}(x)=\\gamma (\\frac{x-\\mu(x)}{\\sigma(x)})+\\beta$$

$\\gamma, \\beta \\in \\mathbb{R}^C$ are affine parameters learned from data;  
mean ad standard deviation, computed across **batch size and spatial dimensions indenpedently for each feature channel**

$\\mu\_c(x)=\\frac{1}{NHW}\\sum\\limits\_{n=1}^N \\sum\\limits\_{h=1}^H \\sum\\limits\_{w=1}^W x\_{nchw}$  
$\\sigma\_c(x)=\\sqrt{\\frac{1}{NHW}\\sum\\limits\_{n=1}^N \\sum\\limits\_{h=1}^H \\sum\\limits\_{w=1}^W (x\_{nchw}-\\mu\_c(x))^2+\\epsilon}$

## Instance Normalization (IN)

$$\\text{IN}(x)=\\gamma (\\frac{x-\\mu(x)}{\\sigma(x)})+\\beta$$

$\\mu\_{nc}(x)=\\frac{1}{HW} \\sum\\limits\_{h=1}^H \\sum\\limits\_{w=1}^W x\_{nchw}$  
$\\sigma\_{nc}(x)=\\sqrt{\\frac{1}{HW} \\sum\\limits\_{h=1}^H \\sum\\limits\_{w=1}^W (x\_{nchw}-\\mu\_{nc}(x))^2+\\epsilon}$

While `BN` takes average among channels, `IN` takes average in each channels. Thus, each channels won't be affected. Image generation is very dependent on channels compared to image classification. `IN` takes very important parts in image generation. State-of-the-art such as CycleGAN, StarGAN ... uses `IN` instead of `BN`.

> In my opinion, `BN` is good for discrminative job and `IN` for generative job.

Here's difference between `BN` and `IN`  
![](https://tmmsexy.s3.amazonaws.com/imgs/2018-04-06-075151.jpg)

## Conditional Batch Normalization (CBN)

First instoduced from [Modulating early visual processing by language](https://arxiv.org/pdf/1707.00683.pdf?ref=sudormrf.run).  
in NIPS2017\. `CBN` is introduced from Auron Courville's lab.

![Pasted image](https://dynalist.io/u/oqmDSTTlNweF4AvRkBCXD63a)

> Also predict delta value of $\\mu$ and $\\sigma$ on `BN`. Thus, $\\mu$ and $\\sigma$ of `BN` will be conditioned to some other Neural Net (questions, query, ...).

## Conditional Instance Normalization (CIN)

$$\\text{CIN}(x;s)=\\gamma^s (\\frac{x-\\mu(x)}{\\sigma(x)})+\\beta^s, s \\in {1,2,3,...,S}$$  
Surprisingly, the network can generate images in completely different styles by using the same convolutional parameters but different affine parameters in IN layers.

> `CIN` is would be good for conditional image generation (sytle transfer for given style. Compute style and use the $\\mu$ and $\\sigma$ for image generation.)

Ulyanov et al. \[52\] attribute the success of `IN` to its **invariance to the contrast of the content image**. However, `IN` takes place in the feature space, therefore it should have **more profound impacts than a simple contrast normalization in the pixel space**. Perhaps even more surprising is the fact that the **affine parameters in `IN` can completely change the style of the output image**.

##### Adaptive Instance Normalization (AdaIN)

`AdaIN` has no learnable affine parameters. Instead, it adaptively computes the affine parameters from the style input:  
$$\\text{AdaIN}(x;s)=\\sigma(y) (\\frac{x-\\mu(x)}{\\sigma(x)})+\\mu(y)$$  
in which we simply scale the normalized content input with $\\sigma(y)$, and shift it with $\\mu(y)$.

> In my opinion, we already use the term `Conditional Normalization` instead of `Adative Normalization`.

- [codeblock in github1](https://github.com/naoto0804/pytorch-AdaIN/blob/master/function.py?ref=sudormrf.run)
- [codeblock in github2](https://github.com/leehomyc/Style-Transfer-Adaptive-Instance-Normalization/blob/master/AdaptiveInstanceNormalization.py?ref=sudormrf.run)

# Applications

## FiLM: Visual Reasoning with a General Conditioning Layer

by [Ethan Perez](https://arxiv.org/find/cs/1/au:+Perez%5FE/0/1/0/all/0/1?ref=sudormrf.run), [Florian Strub](https://arxiv.org/find/cs/1/au:+Strub%5FF/0/1/0/all/0/1?ref=sudormrf.run), [Harm de Vries](https://arxiv.org/find/cs/1/au:+Vries%5FH/0/1/0/all/0/1?ref=sudormrf.run), [Vincent Dumoulin](https://arxiv.org/find/cs/1/au:+Dumoulin%5FV/0/1/0/all/0/1?ref=sudormrf.run), [Aaron Courville](https://arxiv.org/find/cs/1/au:+Courville%5FA/0/1/0/all/0/1?ref=sudormrf.run)  
AAAI 2018\. Code available at this http URL . Extends arXiv:1707.03017.

This work outperforms Deepmind's "relation network" VQA task.

[github](https://github.com/ethanjperez/film?ref=sudormrf.run)

## Conditional Instance Normalization used from "A LEARNED REPRESENTATION FOR ARTISTIC STYLE"

Vincent Dumoulin & Jonathon Shlens & Manjunath Kudlur, **Google Brain**  
![Pasted image](https://dynalist.io/u/qTfFm0m8pyxW_z4DA-mH6VU9)

Outputs multiple style trasfered output with a single network.

## Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data

by **Amjad Almahairi**, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, **Aaron Courville**

Submitted to ICML2018, [arXiv:1802.10151v1](https://arxiv.org/abs/1802.10151?ref=sudormrf.run)

Uses `CIN` for many-to-many mapping.