Neural Style Transfer: A Paradigm Shift for Image-based Artistic Rendering?

Description:

In this meta paper we discuss image-based artistic rendering (IB-AR) based on neural style transfer (NST) and argue, while NST may represent a paradigm shift for IB-AR, that it also has to evolve as an interactive tool that considers the design aspects and mechanisms of artwork production. IB-AR received significant attention in the past decades for visual communication, covering a plethora of techniques to mimic the appeal of artistic media. Example-based rendering (EBR) represents one the most promising paradigms in IB-AR to (semi-)automatically simulate artistic media with high fidelity, but so far has been limited because it relies on pre-defined image pairs for training or informs only low-level image features for texture transfers. Recent advancements in deep convolutional neural networks showed to alleviate these limitations by separating image contents from style, thus making a generalized style transfer practicable. We categorize style transfers within the taxonomy of IB-AR, then propose a semiotic structure to derive a technical research agenda for NSTs with respect to the grand challenges of NPAR. We finally discuss the potentials of NSTs, thereby identifying applications such as casual creativity and art production.

Paper download: (29.6 MB)

Reference:

Amir Semmo, Tobias Isenberg, and Jürgen Döllner (2017) Neural Style Transfer: A Paradigm Shift for Image-based Artistic Rendering? In Holger Winnemöller and Lyn Bartram, eds., Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering (NPAR as part of Expressive, July 29–30, Los Angeles, CA, USA). ACM, New York, article no. 5, 13 pages, 2017.

BibTeX entry:



@INPROCEEDINGS{Semmo:2017:NST,
  author      = {Amir Semmo and Tobias Isenberg and J{\"u}rgen D{\"o}llner},
  title       = {Neural Style Transfer: A Paradigm Shift for Image-based Artistic Rendering?},
  booktitle   = {Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering (NPAR as part of Expressive, July 29--30, Los Angeles, CA, USA)},
  OPTeditor   = {Holger Winnem{\"o}ller and Lyn Bartram},
  year        = {2017},
  pages       = {5:1--5:13},
  articleno   = {5},
  numpages    = {13},
  publisher   = {ACM},
  address     = {New York},
  doi         = {10.1145/3092919.3092920},
  doi_url     = {https://doi.org/10.1145/3092919.3092920},
  oa_hal_url  = {https://hal.science/hal-01527495},
  url         = {https://tobias.isenberg.cc/p/Semmo2017NST},
  pdf         = {https://tobias.isenberg.cc/personal/papers/Semmo_2017_NST.pdf},
}

This work was done in collaboration with the Computer Graphics Systems group at the Hasso Plattner Institute in Potsdam, Germany.