Models and embeddings

My love for Stable Diffusion has been rekindled. So much has been happening and I need to get caught up.

For example, I’m learning how to create and use aesthetic gradients and textual embeddings. These don’t change the model. Instead they encode complex concepts into your inputs. The text prompt input into an AI image generator is transformed into a set of numeric tokens and this is what the textual embedding or aesthetic gradient changes.

I can pull images from a searchable index of the LAION dataset to train the embedding or gradient with a specific concept I want it to understand.

I’ve been getting some good results creating specific types of coloring book images.

I can also download embeddings and models from Civitai. Not all models are safe for work, but the site blurs out NSFW images unless you opt in. Here’s an example of just how much of a difference it makes to use a different model.

Sample Image, standard Stable Diffusion 1.5

Pretty nice, but her robe is on fire and she has a mix between a sword and a strange carpenter’s tool.

Sample Image, Protogen 2.2

After viewing this image I found myself dropping to one knee to propose marriage. The only complaint I have is the right-hand thumb.

Sample Image, Protogen 3.4

Also, not bad. The moon is a bit squashed and the sword hand needs some work.

Even when I generate images of “coloring page illustrations of cats in Tudor period clothing,” the Protogen models are far outstripping the standard Stable Diffusion model they were based on. My next step is to figure out how to do some advanced training of the models myself. My goal is to eventually create a good coloring page model that will have a consistent style I can use to generate unique images that can’t be easily recreated.