Turn a Drawing or Statue Into a Real Person with Stable Diffusion and ControlNet (Ep. 4)

In this guide, you'll learn how to transform a drawing or image of a statue into a realistic-looking person.

Written By: Holostrawberry
Last updated: 2023-06-30

We're going to transform a statue of a man into a real-looking person. Hopefully you are excited, because we are going to make this transformation happen:

Before we start, we want to share that this guide belongs to a series of guides that help you master ControlNet. If you want to skip around, you are welcome to. You can see the list of all guides here:

Now, without further ado, let's get started.

Turn a Drawing or Statue Into a Real Person with Stable Diffusion and ControlNet (Ep. 4)

Did you know you can use more than one ControlNet at the same time? Well, you can, and I'll show you how. Let's begin with this statue example:

First, let me present the problem - I can generate an image using the Normal method, which maps 3D objects in our image:

That's nice! However, the small details are lost. So, one might try the Canny method, which extracts the contours of the image:

Now the details are correct, but the general composition has been lost. So what if we could combine them? We can!

I will add our input image to 2 separate ControlNets, set the first to Normal and the second to Canny. I will increase the Weight of both to 1.5 so that they affect the image more. We can also define when that ControlNet will activate - We could just leave it as is (from 0 to 1, that is, from start to end), but I have determined that Normal is important in the early steps of the generation, while Canny helps later on with all the details.

So, I set the start and end so that they meet at 0.4. That is, after 40% of the steps, Normal will turn off and Canny will turn on. You could try an infinite number of possible combinations, but this is what I'll go with this time. Now, that might sound complicated, but hopefully this short clip will make it clearer:

For reference, here are the rest of my parameters:

And from the 2 ControlNets combined I get this:

I think that's the best of both worlds. We could also improve this with Hires. fix. Just make sure the seed of the image is the same when you want to hires-fix a previous generation.

Not too bad - the right eye is scuffed, but nothing a little photoshop or inpainting can't fix. Here's all of them side to side:

How about we put this man back into our original statue? I won't show you every step of the process, as it depends a lot on your skill level with image editing tools, but here's what I'm gonna do:

  1. Rotoscope (cut out) the man in the final image as best as I can, and put it on top of the original image. It doesn't have to be perfect.
  2. Pass this rough combined image through img2img on Stable Diffusion, at a low denoising level, to smooth out the boundary and any imperfections.
  3. Now we have a decent combined image, but some things will have changed that I wanted to keep. So from this new image I will cut out the boundary between the man and the background, and smooth it out to use as a transition between the original and final images. This is called a "mask".
  4. Put these 3 layers together (shown in the following gif)

This is just one way to do it, but you may find this technique useful when editing other Stable Diffusion generations! We hope this was helpful. And you can check out our other ControlNet guides here:

If you liked this content, found an issue, or would like to contribute, contact us on Discord.