okuha-logo-small-white

Checkpoint Models In Stable Diffusion – The Complete Overview

checkpoint-models-in-stable-diffusion-feature-image

If you click on a link and make a purchase, I may receive a small commission. As an Amazon affiliate partner, I may earn from qualifying purchases.
Read our disclosure.

What Is A Checkpoint Model?

A checkpoint model is a pre-trained Stable Diffusion weight, also known as a checkpoint file (.ckpt). A CKPT file is a checkpoint file created by PyTorch Lightning, a PyTorch research framework. The checkpoint – or .ckpt – format stores and saves models. The .ckpt file contains the entire model, typically several GBs in size.

A Checkpoint model is used to create AI (artificial intelligence) art, and the output depends on the training dataset on which the checkpoint model was trained.

The checkpoint model can’t generate cat images if the training dataset does not include images of a cat or cats.

Example of a checkpoint model:

checkpoint-models-in-stable-diffusion-SDXL-image-results-showcase
SDXL performance showcase. Image credits.

SDXL (Stable Diffusion XL) is a latent diffusion model (.ckpt file) for text-to-image synthesis and is the latest generative model developed by Stability AI (the world’s leading open-source artificial intelligence company). SDXL demonstrates significantly improved performance and competitive results compared to other image generators.

What is Fine-tuning?

Fine-tuning is a machine learning technique where a pre-trained model is further trained on a specific dataset. It allows the model to generate images similar to the narrow dataset while retaining the general capabilities of the original model.

While the original Stable Diffusion versions 1.5, 2.1, etc., were already trained with an extensive dataset, the models lack when it comes to specific art styles or themes. For this reason, you can find thousands of user-generated checkpoint models from sites like Civitai that are fine-tuned with specific themes, ideas, concepts, art styles, or aesthetics in mind.

Difference Between Stable Diffusion Base Model And Fine-tuned Model

checkpoint-models-in-stable-diffusion-SD-base-model-example
Image showing how a fine-tuned checkpoint model is based on a version of Stable Diffusion. For example, the majicMix checkpoint model is based on Stable Diffusion v1.5.

The Stable Diffusion (SD) base model is trained with a massive dataset, such as LAION-2B-EN or similar. The training takes a lot of computational resources and is a somewhat complex process. The SD base model is, therefore, a base that is a general-purpose AI image generator.

SD can be good at everything but not excellent. For this reason, users started to fine-tune the base model for it to be more capable of producing specific types of images. Fine-tuned models are an improved version of the base model released by Stability AI, fine-tuned with a significantly smaller dataset.

While the checkpoint models (found online) are named, for example, Realistic Vision, GhostMix, etc., the base model they are built on is usually some version of Stable Diffusion.

Check out: Best Stable Diffusion models

How to Create Checkpoint Models?

Checkpoint model creation consists of UNet parameters, transformer blocks, pre-trained text encoder, and pooled text embedding, among other techniques.

Creating a checkpoint file or a checkpoint base model requires a lot of computational resources and a massive training dataset. LAION is one of the biggest contributors to dataset development.

Stability AI is one of the most significant contributors to releasing and building base checkpoint models.

Stability AI has so far released Stable Diffusion 1.5, 2.0, 2.1, and SDXL (0.9, etc.), all of which are base checkpoint models that can be fine-tuned (some license restrictions apply to the SDXL base model) further.

How to Fine-tune A Checkpoint Model?

There are two main techniques for fine-tuning a checkpoint model, additional training or using the Dreambooth extension. These methods involve starting with a base model like Stable Diffusion v2.1 or v1.5.

Additional training

Additional training involves training a base model, such as Stable Diffusion v2.1, with an additional dataset focusing on a specific area of interest. For example, you can train the model with a supplementary dataset of dog images to bias its generation towards the aesthetics of this particular dog sub-genre.

Dreambooth

In 2022, the Google research team introduced Dreambooth, a technique to fine-tune diffusion models, specifically Stable Diffusion, by incorporating a custom subject into the model

Using a few input images (typically 3-5) and the corresponding class name (e.g., “A photo of a [V] dog”), the Dreambooth method creates a personalized text-to-image model that encodes a unique identifier ([V] in this case) for the subject.

This identifier can be implanted in different sentences during inference to synthesize subjects in diverse contexts.

Where to Find Checkpoint Models?

The biggest sites for checkpoint models (fine-tuned) are Civitai and Hugging Face. From Hugging Face, you can find Stable Diffusion base models and user-generated checkpoint models. From Civitai, you can mainly find user-generated checkpoint models as well as LyCORISs, Textual Inversions, LoRAs, etc.

The majority of the checkpoint models in Civitai are based on some version of Stable Diffusion or a merge of multiple user-generated models (checkpoint merge models).

How to Use Checkpoint Models?

You must install a WebUI (Web user interface) to use checkpoint models. The most popular WebUI is AUTOMATIC1111.

WebUI Installation And Configuration For Windows

To run AUTOMATIC1111’s WebUI and checkpoint models, you need the following files:

automatic1111-download-github
From the Code button, you can see the HTTPS address for the GitHub project.

Note:

  • When installing Git for Windows, use the default settings and options.
  • When installing Python, remember to select “Add python.exe to PATH” at the beginning of the installation.
  • Video tutorial for installing Stable Diffusion.

Downloading and installing checkpoint models

The easiest way to download the checkpoint model is to go to Civitai and filter the results to only include checkpoint models. Be sure to download only models that are safetensors -files (.safetensors -file format), as some .ckpt files can have malicious code in them.

ModelDirectory/FolderFile typesHow to use in prompt
Checkpoints*\stable-diffusion-webui\models\Stable-diffusion*.ckpt, *.safetensors(select a checkpoint model from the upper left-hand corner of the Web UI)
Hypernetworks*\stable-diffusion-webui\models\hypernetworks*.pt, *.ckpt, *.safetensors<hypernet:filename:multiplier>
LoRA*\stable-diffusion-webui\models\Lora*.pt<lora:filename:multiplier>
Textual Inversion*\stable-diffusion-webui\embeddings*.pt, *.safetensors, imagesembedding’s filename
LyCORIS*\stable-diffusion-webui\models\LyCORIS*.safetensors<lyco:filename:multiplier>
Table showing models, where to copy them, and how to use them.

When you’ve placed the checkpoint model in the appropriate folder, run AUTOMATIC1111 WebUI.

automatic1111-webui-interface-guide
Number 1. in the image shows where you can select different checkpoint models for AI art generation.

After you’ve selected a model, you are ready to create AI art with that specific checkpoint model.

Checkpoint Merge Models

Checkpoint merge models are models that merge two or more checkpoint models. The benefit of checkpoint merge models is that each model carries a set of fine-tuned aesthetics to them, and by combining them together, you can generate truly phenomenal images with them.

The key to creating a good checkpoint merge model is to use models that are aligned with each other. Combining the same kind of artistic aesthetic only enriches the output of the diffusion model.

You can create checkpoint merge models directly from AUTOMATIC1111s WebUI.

checkpoint-models-in-stable-diffusion-checkpoint-merge-how-to
You can create checkpoint merge safetensors -file from AUTOMATIC1111s Checkpoint Merger tab.

From the Checkpoint Merger tab, you can select the primary, secondary, and tertiary models to create a checkpoint merge.

It’s preferred that you use safetensors as the checkpoint format. If you have a custom VAE or have experienced that certain VAE produces good results, you can also Bake in VAE from the UI (user interface).

When you’ve made your desired selections, click Merge to begin the merge operation. The merged file can be found in the following folder: *stable-diffusion-webui\models\Stable-diffusion

How to Make Money With Checkpoint Models?

It’s questionable to sell checkpoint models as the dataset used to train the model in the first place includes copyrightable material. However, many services offer AI art generation using the checkpoint models users offer worldwide.

One of these services is Mage.space, which currently operates based on user-generated models.

The money is made by offering computational resources for AI image generation.

Feature image credits.

Search
artist-profile-picture-avatar

Okuha

Digital Artist

I’m a digital artist who is passionate about anime and manga art. My true artist journey pretty much started with CTRL+Z. When I experienced that and the limitless color choices and the number of tools I could use with art software, I was sold. Drawing digital anime art is the thing that makes me happy among eating cheeseburgers in between veggie meals.

More Posts

Thank You!

Thank you for visiting the page! If you want to build your next creative product business, I suggest you check out Kittl!