Introduction to Image Manipulation and Remixing with ControlNet (Ep. 1)

In this guide series you'll learn a variety of ways in which you can boost your creative workflow with the latest AI technologies.

Written By: Holostrawberry
Last updated: 2023-06-30

This is the first part of a series: "How to Master ControlNet". In this guide, we'll introduce you to ControlNet and how to install it. This is foundational knowledge, but if you wish to skip ahead to some of the other guides in this series, you can do so here:

ControlNet can turbocharge anyone's creative workflow -- artist or enthusiast alike. If you are still new to Stable Diffusion, then you should check out our previous tutorial which explains how to generate images using Stable Diffusion.

What is ControlNet?

If you have tried img2img before, ControlNet is pretty similar, but it is often more effective as well as more versatile. Whereas img2img morphs an existing image into a new one, ControlNet creates a brand new image and constrains the result. Both methods have their own uses, but the results can be mind-blowing!

ControlNet is not a single tool, but rather a toolbelt of a dozen or so different techniques, which are divided into two steps: The preprocessor and the model.

  • The preprocessor prepares your image so that the model can parse it correctly.
  • The model analyzes the features of the preprocessed image and constrains the Stable Diffusion generation to produce a similar image to what it was given.

These methods vary widely, for example, the Canny technique will follow the visible contours of the input image, while Depth will approximate a depth map of the input image and attempt to produce a similar 3D environment. Here are some of the common methods and what you might use them for:

Method NamePossible Use-Case
CannyCopy most details
DepthMake a similar 3D environment
NormalMake a similar 3D object
OpenPoseCopy human poses
Scribble & LineartFinish a sketch for an artwork
SegCopy the general elements and composition
ReferenceCopy the style or feel

Installing ControlNet

If you're using the colab from our Getting Started Guide, ControlNet is already installed. However, if you're running a local installation you may need to install it manually. Skip ahead if you know you already have it.

First, start your Automatic1111 WebUI, then navigate to Extensions -> Install from URL, paste the link to the ControlNet extension, and press the Install button. Then, restart the entire program.

However, you must also download the individual ControlNet models you wish to use. Find them here. Place the model files in the folder /extensions/sd-webui-controlnet/models, inside the main WebUI installation folder.

Then, you will find ControlNet near the bottom of the txt2img and img2img tabs, as a dropdown you can open. It will look like this:

Now with that background, let’s get into some practical examples of using ControlNet. You can start with the next episode (ep.2), or you can skip to whatever content you'd like

If you liked this content, found an issue, or would like to contribute, contact us on Discord.