okuha-logo-small-white

LoRA In Stable Diffusion – Everything You Need to Know

LoRa-in-stable-diffusion-feature-image-v2

If you click on a link and make a purchase, I may receive a small commission. As an Amazon affiliate partner, I may earn from qualifying purchases.
Read our disclosure.

What Is LoRA?

LoRA (Low-Rank Adaptation) is a method published in 2021 for fine-tuning weights in CLIP and UNet models, which are language models and image de-noisers used by Stable Diffusion. LoRAs modify the output of Stable Diffusion checkpoint models to align with a particular concept or theme, such as an art style, character, real-life person, or object.

While LoRAs can be used with any Stable Diffusion model, sometimes the results don’t add up, so try different LoRA and checkpoint model combinations to get the best results. LoRA models can’t be used alone, so they always need a checkpoint model to work.

The usual file size of LoRAs is 10-500MB, compared to checkpoint models that span from 1 to 7GB.

LoRA examples:

Gacha splash LORA
Yae Miko | Realistic Genshin LORA
Anime Lineart / Manga-like Style
Urban Samurai | v0.14 | Clothing LoRA

Check out: LyCORIS in Stable Diffusion

Where to Find LoRA Models?

LoRA models are available in open-source repositories like Civitai and Hugging Face. You can use these models freely but do make sure they are safetensors-files.

How to Use LoRA With AUTOMATIC1111?

When you’ve downloaded a LoRA to your computer, add the model to the following folder: *\stable-diffusion-webui\models\Lora

To add LoRA to the text prompt, the format <lora:filename:multiplier> should be used, where “filename” represents the name of the file (you previously downloaded) containing LoRA (excluding the extension), and “multiplier” is a number typically ranging from 0 to 1, determining the strength of LoRAs impact on the output. LoRA cannot be added to the negative prompt.

Note: Only the LoRA from the first prompt will be used if multiple prompts are used in a batch.

However, it’s important to note that the current version of AUTOMATIC1111’s Web UI does not support LoRA networks for Stable Diffusion 2.0+ models.

Add the LoRA model to the text prompt to activate the LoRA, for example:

your normal text prompt, 1 girl, best quality, <lora:YaeMiko_Test:1>

lora-example-wiht-and-without
Both images were created with the same prompt, except the right side image used Yae Miko LoRA (<lora:YaeMiko_Test:1>)

You can also use trigger words (usually given with the LoRA model) with a specific LoRA to emphasize the look you are after.

LoRA Model Styles

There are various LoRA models available, such as style, concept, character, etc. LoRAs.

For the model style test, I used more or less the same text prompt for each image generation while changing the LoRA. This test showcases how the LoRA affects the AI image generation output.

anything-v5-defaul-result

Model used: AnythingV5

LoRA used: none

LoRA model style: default

anything-v5-defaul-result-line-art-lora

Model used: AnythingV5

LoRA used: animeoutlineV4_16

LoRA model style: Art style (line art)

anything-v5-defaul-result-makima-lora

Model used: AnythingV5

LoRA used: makima_offset

LoRA model style: Character style (Makima from Chainsaw Man)

anything-v5-defaul-result-crystal-lora

Model used: AnythingV5

LoRA used: CrystallineAI-000009

LoRA model style: Concept style (crystals)

anything-v5-defaul-result-clothing-lora

Model used: AnythingV5

LoRA used: urbansamurai_v0.3

LoRA model style: Clothing style (urban, tech wear)

anything-v5-defaul-result-pose-lora

Model used: AnythingV5

LoRA used: Shirt Tug Pose

LoRA model style: Pose style (shirt Tugging)

Art Style LoRA

Art style LoRA focuses on capturing artistic styles rather than specific characters, poses, or objects. These models are trained on the signature styles of artists, enabling users to incorporate those styles into their own artwork.

Using style LoRA, images can be stylized or created in unique artistic styles or artist styles such as animated looks, watercolor paintings, Greg Rutkowski, Studio Ghibli, oil paintings, and line art. The advantage of style LoRAs is their compatibility with regular Stable Diffusion checkpoints, allowing you to combine different styles and create exceptional pieces without the need for merging large models.

For instance, by combining a Realistic Vision checkpoint with a line art style, AI can generate realistic images in a line art style.

Concept LoRA

Concept LoRA is a model trained on specific concepts or ideas. These models aim to conceptualize abstract concepts such as emotions, materials, actions, etc. You can use concept LoRA when creating AI images that aim to convey a specific concept.

In my example (above), I used the CrystallineAI LoRA model to add crystal effects and concepts to the image. LoRAs can help you generate certain elements to an AI image that are specialized in a certain style, theme, concept, etc.

Character LoRA

Character LoRA refers to a type of model specifically trained on individual characters from various media, such as cartoons, video games, and anime. This type of LoRA excels at capturing a character’s unique appearance and defining features, making it a popular choice for generating AI illustrations, character concept art, and reference sheets.

Character LoRA allows for a quick generation of authentic-looking characters, with some models even enabling customization of outfits and settings. These models cover many characters from popular franchises like Chainsaw Man, Genshin Impact, DC, Super Mario, Marvel, Pokémon, etc.

You can create pretty awesome results by combining a character LoRA with a certain art style LoRA.

Pose LoRA

Pose LoRA helps you achieve a certain pose without extensively describing it to the AI art generator. There are not that many pose LoRAs compared to art styles or character LoRAs. Pose styles are useful when you want to retain a certain concept, art style, etc. but want the character’s pose to be something specific.

Clothing LoRa

Clothing LoRAs are specifically trained with a certain clothing style or fashion in mind. You can easily add specific clothing styles like urban, mechanical, Blade Runner-esque, Chinese, historical, streetwear, etc., to your character without writing a lot of detailed words to the text prompt.

Clothing LoRA tends to keep the character style, features, and details intact while applying a certain clothing style or fashion for your character.

Object LoRA

Object LoRA refers to a diverse category of LoRA models used for generating various objects, ranging from furniture, kitchenware, and plants to vehicles. You can also use object LoRAs to generate UI elements, video game items and assets, or elements for websites.

The specific items that can be created depend on the chosen model and the text prompt you use.

How Does LoRA Work?

LoRA-stable-diffusion-working
Image showing the Diffusion process and the denoising U-net with the cross-attention mechanism. Image credits.

LoRA is a special technique that helps make certain parts of models smaller without losing their effectiveness. It focuses on the cross-attention layers (see the image above), which are like spreadsheets with lots of numbers. LoRA breaks these spreadsheets into smaller parts to save space while keeping the important information intact.

These models gradually remove noise from a randomly generated variable to create realistic images. They use a compressed latent space, which is a simplified representation of the image, to generate new images efficiently.

Unlike previous methods, they use convolutional layers and focus on important details to improve image quality. The model consists of an encoder (E) and a decoder (D) that work together to generate images.

Check out for more information: High-Resolution Image Synthesis with Latent Diffusion Models

How to Train A LoRA?

To train LoRA effectively, the recommended approach is to use kohya-ss, Koyha is a user interface for training models. Stable Diffusion web UI now supports LoRA trained by sd-scripts.

Normally, LoRA training needs a minimum of 6 GB of VRAM. LoRAs are trained with a dataset that has an image and a .txt file associated with it. The image and the .txt file need to have the same name. For example, 1.png and 1.txt. The text (txt-file) is a text prompt of the image in question, though there are other methods to this too.

The training takes about 5 hours on a 2080 Ti GPU with 11GB of VRAM.

You can read more about how to train LoRA through HuggingFace.

Advancements In The Field of LoRA

There have also been advancements in creating LoRAs more efficiently called Generalized LoRA (GLoRA). GLoRA is a novel approach for fine-tuning large-scale pre-trained models that achieves remarkable results in enhancing fine-tuning and transfer learning abilities.

GLoRA leverages a generalized low-rank adaptation and re-parameterization framework, effectively reducing the parameters and computational requirements for fine-tuning. This makes GLoRA a practical and resource-efficient method for real-world applications.

This research contributes to improving the fine-tuning process for large-scale pre-trained models and paves the way for future investigations. These include further exploration of generalized low-rank adaptation techniques, hybrid approach development, and search and optimization algorithms refinement.

Feature image credits.

Search
artist-profile-picture-avatar

Okuha

Digital Artist

I’m a digital artist who is passionate about anime and manga art. My true artist journey pretty much started with CTRL+Z. When I experienced that and the limitless color choices and the number of tools I could use with art software, I was sold. Drawing digital anime art is the thing that makes me happy among eating cheeseburgers in between veggie meals.

More Posts

11 Best Anime Stable Diffusion Models for Anime Art Generation

8 Best AI Anime Art Generators – Next-Level Anime Art Generation

My Honest PixAI Review – Pricing, Features, Use Cases

How to Use AUTOMATIC1111 WebUI – Full Beginners Guide

AI Art Models – Everything You Need to Know (Incl. SD Models)

How to Use Stable Diffusion For AI Art Generation – Beginners Guide to AI Art

Contact and Feedback

Thank You!

Thank you for visiting the page! If you want to build your next creative product business, I suggest you check out Kittl!