Cover image prompt: "A cyberpunk engineering human with turquoise tones talking with a robot representing AI - cinematic."

Prompt engineering is how we communicate our ideas and receive the desired results from an AI model, like Stable Diffusion (SDXL). In this blog, we will learn how to enhance the accuracy and effectiveness of the text-to-images generated with SDXL.

AI tools are becoming more popular by the day, and prompt engineering is a helpful skill one should learn to master. Generative art using models like SDXL is evolving incredibly fast. AI art, or generative art, requires processes and skills different from traditional art. AI has rapidly come at us, and it is not going away. Learning how to communicate with artificial intelligence algorithms is crucial to staying ahead of the curve, working with this medium, and producing more desired outputs.

Prompting = Instructions + Context

In the context of prompt engineering, the most important thing to keep in mind is clarity and specificity.

Most of us understand that we need to provide instructions to an AI to get an output, but how we specify these text-based prompts and how detailed they are will influence our outputs.

Example prompt: "A cyberpunk engineering girl with turquoise tones working with complex equations and computer instructions all around surrounding her - cinematic."

"A machine learning model is only as good as the data it is fed" - Reynold Xin.

The Quality of Data Input Matters

Prompt engineering stands between the intersection of art and science, a craft of instructions to steer AI. Be specific, think about your desired outcome, and clearly define prompts. Crafting the perfect prompt is an art that includes clarity and specifics to effectively guide these machine learning models in delivering.

These text-to-image generation models need help understanding the exact meaning behind these text-based prompts. To effectively communicate our ideas, we must communicate the idea clearly and produce a highly descriptive prompt.

• Make it easy for the model to understand what you are imagining.

• Specific and highly descriptive details will narrow down the generated outputs.

• Text prompts that are dense in specifics and descriptive language will ensure that the model stays on course and returns a sound output. Feeding too simple instructions can give your AI too much to choose from, ultimately leading it to take an unexpected course and produce inaccurate or odd images.

Balance Conciseness with Detail

Be descriptive and concise. It's crucial to be as descriptive as possible. Add various styles, colors, backgrounds, and descriptive words to your prompt. The more descriptive, the better, but don't overdo it.

Specify Art Style.

Reference a known artist like Van Gogh or any art form and style you are going for. Capturing the medium using phrases like painting, photograph, and cartoon character.

Example prompt: "A girl anime character looking at an aquarium in the style of Studio Ghibli."

Adding Color and Shade

Describe the light, shade, and mood of your image to add variety, using phrases like dark lighting, sunny, bright, or gloomy.

Example prompt: "A bright and purple cat walking down a dark alley in the rain."

Finding the Middle Ground of a Good Prompt

Using straightforward language that minimizes the chance of misinterpretation is finding the right balance. Clearly choosing your words to describe your idea is key, but overdoing it can make your prompt confusing.

How do you find the right balance between too simple - and too much detail? Let's see some examples.

Refine and Iterate to Improve Your Prompt Engineering.

A great way to learn and perfect the build of your prompts is to refine your work and iterate multiple times. Like any skill, the more you practice, the better your prompt anatomy will become.

Example prompt: "Multicolored rainbow eruption of color aura flow universe"

Over time, testing adding detail to your prompts and keywords that distinguish your thoughts to the models can help you find the perfect prompts, what to avoid, and specific word arrays to perfect the skill of prompt engineering.

Bringing it back to Lilypad

With Lilypad, when we run an SDXL pipeline module, several additional inputs to enhance images are available for you to use. Based on ComfyUI, the SDXL Pipeline modules for Lilypad allow you to generate images using Stable Diffusion XL and related models. They support both the Refiner and Base versions of SDXL v0.9 and v1.0.

These modules are designed to be run in a Docker container, either through the Lilypad Network or directly in Docker. To refine your images, you can customize various parameters such as prompt, seed, steps, and more.

For a clearer understanding of how to run the SDXL Pipeline in Lilypad, here are some example commands:

SDXL v0.9

Base: "lilypad run sdxl-pipeline:v0.9-base-lilypad3 -i Prompt="an astronaut floating against a white background"

SDXL 1.0

Refiner: "lilypad run sdxl-pipeline:v1.0-refiner-lilypad3 -i Prompt="an astronaut floating against a white background"

For more information and examples, visit the Lilypad SDXL Pipeline module repository.

Conclusion

Generative art through AI models like SDXL is a craft of its own. It involves different techniques, artistic choices, practice, and decisions while comparing results—a recipe of trial and error to achieve the text-to-image you envision.

Share your creations with Lilypad! 🪷
Make sure to tag us on Twitter/X @lilypad_tech to share your text-to-image process!