What is synthetic reality (SR)?

The term “synthetic reality” was first coined in 2014 in a talk by Matt White at UC Berkeley to describe how artificial intelligence could generate synthetic versions of real-world objects both in visual representation and physical properties.  Synthetic reality is a blanket term for any synthetic object (modeled on living and non-living entity) and the synthesis of real-world physics and natural phenomena in a synthetic virtual environment.

Artificial Intelligence, specifically generative AI, has become powerful tool for simulating real-world physics and generating both photorealistic and artistic 2D and 3D digital assets as well as allowing creators to transform style, and employ novel techniques such as in and out painting.  The abilities of generative AI continue to evolve at a feverish pace and extend beyond just images and 3D but to speech, text, music, design and so much more.

When it comes to 3D models and rendered assets, state-of-the-art techniques are able to synthesize new 3D digital assets from labeled data (either 2D images and transformed to 3D or trained on 3D assets) but we will soon see novel methods that can successfully employ self-supervised and one-shot/few-shot techniques to train on unlabeled data which will give tremendous creative power to artificial intelligence.

There are a variety of modeling techniques currently employed (Q2/2022) including transformers, generative adversarial networks (GANs), autoregressive convolutional neural networks (AR-CNNs), graph neural networks (GNNs), variational autoencoders (VAEs), diffusion autoencoders, long-short term memory (LSTM) and the list continues to grow with new innovations in the space.

Text and audio to 3D techniques will help enable Metaverse builders to quickly construct 3D assets and scenes with no programming knowledge. A new method of programming has evolved which we now call “prompt engineering” (in 2014 Mr. White used the term wordsmithing) where instead of writing code, creators devise creative descriptions to create unique and 3D digital objects with utility and 3D scenes for world building and Metaverse commerce.

The area of generative AI continues to evolve and will need to be further democratized so that anyone can access and train generative models without having to pay substantial upfront hardware costs or cloud services fees.  The state-of-the-art for image generation and large language models train on billions of image-text labels, which is just not accessible for anyone outside of industry or academia. 

2023 will be a big year for generative AI and we expect to see more and more innovations in applications of neural methods for 2D and 3D asset creation, in animating characters, bringing life to NPCs, training autonomous agents and robots, generating 3D scenes, physics simulation, as well as generative audio, music, text and code.