Art-directing Appearance using an Environment Map Latent Space

Problem: Artists need to make stylistic 3D lighting edits uniquely for different environments, e.g. "boost rim-light when sunny", and "add highlights when cloudy". Current artistic editing techniques can't adapt to different changing lighting environments, which is needed for real-time games and AR apps.

We developed a new workflow for artists to art-direct 3D appearance across different lighting conditions. To do this we learned a latent space of varied lighting environments, that enables the following:

  1. In look-development, artists can intuitively browse the latent space and preview many environments organised in semantic clusters.
  2. Artists then paint directly on the space to specify how 3D lighting edits should vary across environments.
  3. Games and AR apps with dynamic environments, encode them into the latent space, and look-up the desired artistic appearance.

Environment Map Latent Space teaser image

(a) Our environment map (EM) latent space expresses semantic clustering in 2D, for navigating a varied EM database during look development.
(b) Art directing material and lighting edits by 2D painting across the latent space.
(c) Art-directed appearance is preserved in similar EMs, and dynamically adapted to changing environment lighting.

Character model by Emiliano Colantoni

Paper Presentation for PG2021