Justin Pinkney and Doron Adler have been working on a project called Toonify, which can turn a photographic face into a cartoon character. It’s not yet perfected, but the story behind how they did it is kinda neat. The normal way to train an algorithm to do this would involve a large dataset of portraits and their corresponding cartoon characters. Since that does not exist (yet), they had to start from scratch and create such a dataset.
Listen beautiful relax classics on our Youtube channel.
First, Doron Adler trained a StyleGAN model—the same tech behind This Person Does Not Exist, a site which randomly spits out photorealistic people that, as the name implies, are entirely computer generated—on Disney, Pixar, and Dreamworks characters so it could recognize features are quintessentially cartoony. The model then automatically selected fake people from the This Person Does Not Exist universe and augmented them with those cartoon features. But StyleGAN globbed all the styles from computer generates images, cartoons, and photographs together equally, which meant that the same person might have tufts of realistic hair, CGI meatball cheeks, and eerily flat hand-drawn eyes.
This is where Justin Pinkney came in with his model, which they blended with Adler’s.
Pinkney developed a “layer-swapping” process to parse out the desirable characteristics from each image: the cartoon half affects only the structure of the resulting toonified face, while the human half contributes the lighting and other high-resolution details.
There’s a lot more to it, of course. Is there an app where I can make my own face into a cartoon? Yes, but you can’t do it (yet). First, the program doesn’t use your face. It will select a computer-generated photorealistic face from the This Person Does Not Exist library that looks like you -and the program is pretty good at that, although it takes some time. Second, when they launched the online app, so many people invested the necessary time to try it that they couldn’t afford the bandwidth bill, so they took it down. Adler and Pinkney hope to work out the kinks and make it available to the rest of us soon. Read the story at Gizmodo.
(Image credit: Justin Pinkney and Doron Adler)