Image prompting with JSON

The image generator I use, Ideogram, just released a new model. This model is leaning towards precision. It includes a system that uses a JSON format to create detailed prompts. You can feed it an image and generate a complex prompt with a high-level description and compositional breakdown that includes a description of the background, and precisely placed objects and text.

I happened to create a new app yesterday that makes a simple image with 9 emoji to represent your day, My Day in Nine. Claude used its usual beige defaults, so I wanted to see if Ideogram could create alternate designs.

I fed a screenshot of the original into the system and generated a description. It’s 590 lines long so I won’t reproduce it here. But it uses bounding boxes to place everything on screen. Here’s a sample.

     {
        "type": "text",
        "bbox": [
          39,
          252,
          88,
          469
        ],
        "text": "My Day in Nine",
        "desc": "Large headline-sized text at the top
center. Bold serif font in black."
      },

Original Screenshot

With this JSON prompt I can generate a pretty close approximate of the original screenshot. But that’s not the use case I was thinking of. I fed the description to Claude and had it preserve the proportions and text, but had it update the descriptions to make a dark mode. I fed this back into Ideogram and generated a mock-up of a dark mode.

Reimagined Dark Mode

This still isn’t very impressive of a use case. I could have coded this with Claude faster. The real test is trying out something ambitious. So I had Claude rewrite the JSON descriptions to turn it into a fantasy RPG screen. I told it to specifically add things that CSS would struggle to render.

Reimagined Fantasy Screen

This is where it starts to get interesting. In theory I could produce dozens of variations of a website design just by updating the descriptions with an LLM that’s capable of preserving the original structure. Once I find a design I like, this can be fed back to Claude to start re-coding the actual CSS.