Turn your Photos Into Paintings with Stable Diffusion and ControlNet (Ep. 3)

This guide demonstrates how you can turn a photo into a digital painting with ControlNet.

Written By: Holostrawberry
Last updated: 2023-06-30

In this guide, we'll go through an example of how to turn a photo into a digital art painting using ControlNet.

ControlNet can turbocharge anyone's creative workflow -- artist or enthusiast alike. If you are still new to Stable Diffusion, then you should check out our previous tutorial which explains how to generate images using Stable Diffusion.

This is the second part of a series: "How to Master ControlNet". You can explore all guides in this series below:

Now, let's dive in!

Turn your Photos Into Paintings with Stable Diffusion and ControlNet (Ep. 3)

ControlNet can give new life to your photos or artworks, with a variety of techniques. Because it can take the shape of one image to create a new one, we can transfer any style that has been trained into a Stable Diffusion model.

Below is my input image. Since we're working with photos, the images may be quite large. I would recommend that you first resize your image to a maximum of 1024 width or height. In this case, my images will be 1024x768.

I will load the image into ControlNet by opening the dropdown near the bottom of my txt2img page. Drop your image, don't forget to check off Enable, and select the method you wish to use. I'll showcase the Depth method, which is designed to mimic 3D environments. I'll also enable Pixel Perfect since I want my input and resulting image to have the same size. Watch this short clip:

I will use the PastelMix model for this example, as it gives a beautiful watercolor feel to your generations.

I'll use most of the default settings for now, and I will set the resolution to half of my input image (512x384). You'll see why in a bit.

For the prompt I'll write a few simple terms: snow, mountain, night sky, cabin

For the negative prompt I'll only type EasyNegative; this is an embedding which you can add to Stable Diffusion, and it comes included in the colab from the first guide. The purpose of this embedding is to improve the quality of your images by replacing most or all of your negative prompt. If you're not sure how to use embeddings, here's a simple negative prompt you can use instead: (low quality, worst quality:1.4)

These are cool, but I can make it resemble my input image more by increasing the Control Weight (found inside the ControlNet dropdown from earlier). I'll also try to get rid of the trees.

I like that first one! Now, since this image is only half the size of my desired final image, I’m going to enable Hires. fix. First I want to ensure I generate the same image again, that means we'll need the same seed. For that I select the image I liked on the right, then click the recycle button next to the seed box on the left to copy its seed.

I enable Hires. fix, which causes a few more options to appear. The PastelMix model recommends setting the Upscaler to Latent and the Denoising strength to 0.6, so we'll do that.

Here is my final image:

Here's another example. For this one, the Canny and Depth ControlNet types didn't work very well for me, as the AI didn't know that the oval in the middle was supposed to be a lake. So I'm trying the Seg type, which segments our image into its components to copy the general composition.

I haven't shown it to you explicitly yet, but when generating an image with ControlNet we'll also get the preprocessed middle step. In this case, I got this:

Looks good, let's get to it.

Stunning. We could get even cooler results if we used more than one ControlNet at once! But that's it for this tutorial. Make sure to check out the rest of the ControlNet guides below. Cheers!

If you liked this content, found an issue, or would like to contribute, contact us on Discord.