Blog: Getting structured data out of images with Granite Vision 4.1

[planetf1]

What this is showing off

Vision models return prose. The point of this post is that they don't have to.

The blog demonstrates Mellea's extraction pattern — pass a format= Pydantic model to
m.instruct(), get a typed Python object back instead of a string. No JSON prompt
engineering, no json.JSONDecodeError handlers, no post-processing regex. The return type
is the contract, and constrained decoding enforces it.

It then builds up two layers of validation on top:

• requirements= — plain-English semantic constraints (date format, positive totals).
  The model retries with the failed requirement injected into the repair prompt.
• IVR validation_fn — programmatic arithmetic check (line items × quantities = subtotal).
  The failure reason gets fed back into the repair prompt verbatim.

The receipt image is synthetic (PIL-generated) with a thermal-printer smudge over part of
the subtotal, giving the validation layers something realistic to catch.

Strategy fix (latest commit): Both validation sections now use RepairTemplateStrategy
instead of RejectionSamplingStrategy. RejectionSamplingStrategy.repair() returns the
unchanged action/context — same prompt, no feedback. RepairTemplateStrategy builds a
repair prompt with the failed requirement or ValidationResult.reason injected, which is
what the surrounding blog prose describes.

---

Status: Draft — scenario still being refined

The Mellea API usage and code structure are stable. The receipt scenario is still being
iterated. Detection is reliable: the date format requirement consistently detects 22/03/2026
and the arithmetic check confirms extractions are correct. Repair of the date format issue
at 4b scale is not guaranteed (the blog's conclusion now says this explicitly). Receipt values
may change before publication.

---

Model availability — why this is blocked

The blog is written for Ollama, which is the right default for a local-first post. The
problem: Ollama requires GGUF format, and Granite Vision 4.1 is only available as full
bfloat16 safetensors on Hugging Face right now (~8 GB download, not a 4-bit quantized GGUF
like you'd get from ollama pull).

Ollama cannot load safetensors directly — it needs IBM to publish a GGUF to the Ollama
library (or a community conversion to appear). Until then, the testing path is mlx-vlm on
Apple Silicon or vLLM, both of which can serve safetensors directly.

Watch https://ollama.com/library for granite-vision-4.1. When it lands: remove the
editorial note, verify ollama pull granite-vision-4.1 works, flip to ready.

---

Reviewing now

Follow the editorial note at the top of the post. Short version:

1. Set up a clean environment and start the model server:

mkdir granite-vision-test && cd granite-vision-test
uv init --bare --python 3.12
uv add mlx-vlm mellea pillow
uv run python -m mlx_vlm.server --model ibm-granite/granite-vision-4.1-4b

Model downloads ~8 GB on first run (full bfloat16 safetensors — larger than an Ollama pull).
Serves at http://localhost:8080/v1.

2. In each code snippet, swap the session setup from:

m = start_session(model_id="granite-vision-4.1")

to:

m = MelleaSession(OpenAIBackend("ibm-granite/granite-vision-4.1-4b",
                                base_url="http://localhost:8080/v1", api_key="mlx"))

---

Test plan

• npm run dev — confirm post renders at /blogs/granite-vision-structured-extraction
• Receipt image displays correctly (smudge on subtotal visible)
• Code syntax highlighting looks right on all blocks
• Run the code against mlx-vlm per the instructions above

🤖 Generated with Claude Code

[psschwei]

Don't have a strong opinion here, but assuming it takes a while to get the vision model into Ollama should we consider using vllm in the blog instead?

[ajbozarth]

I'll walk through the blog and try it out myself when I have bandwidth, but to start heres a small review from Claude:

---

Code checks out against current mellea source — APIs, imports, signatures, and the RepairTemplateStrategy switch all verify. Front matter and asset are good. Snippet syntax checks pass; live execution skipped (model not in Ollama yet, per the editorial note). de-llmify score 1.

A few inline notes below. Pre-publish blockers (editorial note removal, Ollama availability) are already tracked in the PR description.

[planetf1]

Don't have a strong opinion here, but assuming it takes a while to get the vision model into Ollama should we consider using vllm in the blog instead?

vllm tends to target more the operational audience rather than developer I think but also doesn't have good macOS support. Whilst there's a fork supporting mlx, it doesn't support vision models. I'd actually suggest sticking with the mlx-vlm workaround if the gguf takes a while for mac. But we could add vllm as the linux workaround. Ollama remains the least friction, so will look into likely timescales.

[planetf1]

Thanks @psschwei @ajbozarth for comments. There are still two areas I'm working on

• the likely timescale for ollama support (sometimes new model releases require changes). If long, fallback could be to add vllm for linux users
• refining the actual scenario - I'm not happy with the run-through currently. Needs a bit more work on the scenario, prompts & mellea constructs. (hence draft - but wanted to share the overall idea and get general review)

[psschwei]

vllm tends to target more the operational audience rather than developer I think but also doesn't have good macOS support. Whilst there's a fork supporting mlx, it doesn't support vision models. I'd actually suggest sticking with the mlx-vlm workaround if the gguf takes a while for mac. But we could add vllm as the linux workaround.

Sorry, I misread your original post, thought I read mlx-vllm. Mostly was just trying to see if ollama is a hard requirement or if serving the model another way would be fine.

Ollama remains the least friction, so will look into likely timescales.

My understanding is that it is being actively worked on, but may take a few weeks (apparently getting vision models working with ollama is not very straightforward).

Though, and not saying we should just posing as an idea, what if we target colab for running the code? The vision model is 4B which might fit on a free instance, and if not would only take a few credits (<$1) to run for an hour. It would also reduce the setup headaches even more (simply click a cell or two and everything is up and running).

[ajbozarth]

The updates addressing my previous review look good, but I'm going to hold off on further review until you've finalized this and moved it out of draft

[psschwei]

closing and reopening to get the DCO bot to fire
edit: didn't work 😕