06 - Image Generation: Stable Diffusion, DALL-E, and Midjourney Under the Hood

Introduction: Creating Images from Nothing

Image generation is one of the most spectacular results of generative AI. Describing an image in natural language and seeing it materialize in seconds was science fiction just a few years ago. Today, tools like Stable Diffusion, DALL-E, and Midjourney make it accessible to anyone.

But how do these models actually work? This article disassembles the architecture of Diffusion Models, explains the differences between the main tools, and provides practical prompt engineering techniques for professional-quality images.

How Diffusion Models Work

Diffusion Models are based on a counter-intuitive principle: destroying an image by progressively adding noise, and then training a neural network to reverse the process. If the model learns to remove noise effectively, it can start from pure noise and generate a completely new image.

Forward Process: Adding Noise

In the forward process, you start from a real image and add Gaussian noise in T progressive steps. After enough steps, the original image is completely unrecognizable: only random noise remains. This process is deterministic and requires no training.

Reverse Process: Removing Noise

The reverse process is where the magic happens. A neural network (typically a UNet) is trained to predict the noise to remove at each step. Starting from pure noise, the model iteratively removes noise for T steps, generating a coherent image.

Stable Diffusion: Open Source and Self-Hosted

Stable Diffusion is the reference model for open source image generation. Published by Stability AI in 2022, it's freely downloadable and runnable on consumer hardware. With a GPU with 8+ GB VRAM, you can generate high-quality images locally.

# Image generation with Stable Diffusion and diffusers
from diffusers import StableDiffusionPipeline
import torch

# Load the model (first run downloads ~5GB)
pipe = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,     # Half precision to save VRAM
    variant="fp16",
    use_safetensors=True
)
pipe = pipe.to("cuda")

# Generate an image
prompt = "A serene Japanese garden at sunset, koi pond with lily pads, cherry blossom trees, soft golden light, photorealistic, 8k"
negative_prompt = "blurry, low quality, distorted, ugly, deformed"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=30,    # More steps = better quality (20-50)
    guidance_scale=7.5,        # How closely to follow prompt (5-15)
    width=1024,
    height=1024
).images[0]

image.save("japanese_garden.png")
print("Image generated!")
      

DALL-E: OpenAI's API Solution

DALL-E 3 by OpenAI offers excellent quality through a simple API. It requires no local GPU or complex setup, but has a per-image cost.

# Image generation with DALL-E 3 via API
from openai import OpenAI

client = OpenAI()

response = client.images.generate(
    model="dall-e-3",
    prompt="A futuristic cityscape at night with neon lights reflecting on wet streets, cyberpunk style, dramatic lighting",
    size="1024x1024",      # 1024x1024, 1024x1792, 1792x1024
    quality="hd",          # "standard" or "hd"
    n=1                    # Number of images (DALL-E 3: only 1)
)

image_url = response.data[0].url
revised_prompt = response.data[0].revised_prompt
print(f"Image URL: {image_url}")
print(f"Revised prompt by DALL-E: {revised_prompt}")

# DALL-E 3 pricing:
# Standard 1024x1024: $0.040/image
# HD 1024x1024: $0.080/image
# HD 1024x1792: $0.120/image
      

Comparison: Stable Diffusion vs DALL-E vs Midjourney

Generation Modes

Diffusion models support several generation modes, each with specific applications.

Text-to-Image

The base mode: from a text prompt, generates a completely new image. It's the most widely used mode and the one that captured the public's imagination.

Image-to-Image

Starts from an existing image and transforms it following a prompt. Useful for stylistic variations, quality improvement, or transforming sketches into complete images.

# Image-to-image: transform an existing image
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load starting image
init_image = Image.open("sketch.png").resize((1024, 1024))

# Transform: from sketch to realistic image
result = pipe(
    prompt="Detailed architectural rendering of a modern house, photorealistic, professional photography",
    image=init_image,
    strength=0.75,          # How much to modify the original (0-1)
    guidance_scale=7.5,
    num_inference_steps=30
).images[0]

result.save("house_rendering.png")
      

Inpainting

Inpainting allows modifying specific regions of an image while keeping the rest intact. Useful for removing objects, adding elements, or correcting defects.

Prompt Engineering for Images

Prompt engineering for images follows different rules than text prompting. The typical structure includes subject, style, quality, and technical parameters.

# Image prompt builder
def build_image_prompt(
    subject: str,
    environment: str = "",
    style: str = "photorealistic",
    lighting: str = "natural light",
    quality: str = "8k, highly detailed, sharp focus",
    extra: str = ""
) -> dict:
    """Build a structured prompt for image generation."""
    parts = [subject]
    if environment:
        parts.append(environment)
    parts.extend([style, lighting, quality])
    if extra:
        parts.append(extra)

    prompt = ", ".join(parts)

    # Standard negative prompt for high quality
    negative = "blurry, low quality, distorted, deformed, ugly, watermark, text, signature, cropped, worst quality, low resolution"

    return {"prompt": prompt, "negative_prompt": negative}

# Examples of structured prompts
prompts = [
    build_image_prompt(
        subject="A majestic snow leopard",
        environment="on a Himalayan mountain peak",
        style="National Geographic photography",
        lighting="dramatic sunset backlighting"
    ),
    build_image_prompt(
        subject="A cozy Italian cafe interior",
        environment="narrow cobblestone street visible through window",
        style="warm watercolor illustration",
        lighting="soft warm afternoon light"
    )
]

for p in prompts:
    print(f"Prompt: {p['prompt']}\n")
      

Customization: LoRA and ControlNet

Stable Diffusion offers two powerful mechanisms for customizing generation:

• LoRA for images: trains an adapter on a few images of a specific style or subject (e.g., your face, a brand style). Result: the model generates images in that specific style
• ControlNet: provides an additional control signal to the model: a human pose (skeleton), image edges (canny edge), a depth map, or a segmentation. The model generates the image respecting that structural signal

Copyright and Ethics in Image Generation

AI image generation raises significant legal and ethical questions that cannot be ignored.

Copyright Issues

• Training data: models are trained on billions of images collected from the web, many copyright-protected
• Artistic style: generating images "in the style of [living artist]" is ethically questionable
• Output ownership: who owns the rights to an AI-generated image? Legislation is still evolving

Ethical Best Practices

• Don't use AI to imitate living artists' styles without permission
• Disclose when an image is AI-generated
• Don't generate content that could be used for disinformation
• Respect the usage guidelines of models and providers

Conclusions

AI image generation has opened unprecedented creative possibilities, but requires technical understanding to achieve professional results. Stable Diffusion offers flexibility and total control for those willing to invest in customization. DALL-E 3 is ideal for those seeking immediate quality via API. Midjourney excels in artistic style.

Copyright and ethical questions are real and evolving. As professionals, it's our responsibility to use these tools ethically and respect artists' rights.

In the next article, we'll return to code with Generative AI for Software Development: GitHub Copilot, Claude Code, Cursor, and best practices for using AI assistants in daily programming.