When Gemini final generation failed with 503 UNAVAILABLE, the easy fix would have been to add another retry. But the more important Place2Page question was different: which failures should the product absorb inside the user flow, and which failures should stay visible to operators? I treated only capacity-like errors as eligible, sent them to a fallback provider once, and kept fallback metadata even when the generation succeeded.
Summary
- Problem: Temporary Gemini final HTML generation failures such as
503,UNAVAILABLE, andoverloadedinterrupted the user creation flow. - Decision: Do not hide every failure behind fallback. Only absorb errors that looked like provider capacity issues.
- Fix: Retry once with a fallback model, and record the primary provider, fallback provider, and reason in result metadata.
- Lesson: AI provider failures are not only outages to hide. They are reliability design decisions about which failures a product path should absorb.
Context
Place2Page uses Gemini for final HTML generation. The service fetches place data, builds a meta prompt, and asks Gemini to produce the final HTML in the last step.
Most of the time, that works. But once a generative AI provider sits inside a product flow, one operational condition becomes unavoidable.
Sometimes the external model is busy.
The official Google Gemini API troubleshooting guide describes 503 UNAVAILABLE as a state where the service may be temporarily overloaded or down. In other words, the error does not always mean our code is wrong.
The user does not experience that distinction. They do not think, "Gemini is busy." They think, "My page creation failed."
Problem
At first, model call failures were surfaced as errors. That was honest for debugging, but weak for the user flow.
Final HTML generation happens near the end of the whole creation process. The user has already waited while the service fetched place data, built a meta prompt, and prepared the generation. If the last model call fails after that, the experience is poor.
But falling back on every error would be wrong too.
If the API key is invalid, the request payload is malformed, or the model is called in an unsupported way, fallback would hide a real defect. Later, it would become harder to explain why fallback keeps happening.
So the system needed a boundary.
Initial Hypothesis
I assumed not all failures should be treated the same. The product should absorb only temporary capacity problems inside the user flow.
The rough split was:
- Absorbable failures:
503,UNAVAILABLE,high demand,overloaded - Failures to surface immediately: invalid configuration, authentication errors, bad requests, code bugs
The first group is closer to provider state. The same request may succeed a little later or on another model. The second group is something we need to fix. It should not be covered by fallback.
Evidence
This was not handled by reading one log line and guessing. The tests left in the repo show the intended boundary.
The Gemini stream failure test simulates this error:
503 UNAVAILABLE. {'error': {'message': 'high demand'}}That error is closer to provider capacity than internal validation or prompt parsing. So the generation service carries explicit markers for capacity failures.
503
unavailable
high demand
overloaded
temporarily unavailable
try again laterThe fallback test verifies that the first call fails with 503 UNAVAILABLE: high demand, then the second call moves to gpt-5.3-codex. The successful result keeps fallback metadata.
provider_fallback_applied == True
provider_fallback_reason == "gemini_provider_unavailable"
primary_final_model_provider == "gemini"
final_model_provider == "openai"
final_model_id == "gpt-5.3-codex"There is also a test to make sure not every Gemini error falls back. Without that distinction, authentication errors or invalid requests could be hidden behind fallback.
At the commit level, 267b3fc fix(api): fallback from busy Gemini to OpenAI changed generation_service.py, config, tests, and model routing docs together. It was not a simple retry. It included model routing and operational metadata for later inspection.
Implementation
The final HTML generation path detects errors that look like Gemini capacity failures. If a fallback model is configured, the required API key exists, and the fallback is not the same model as the primary, it retries once with the fallback model.
The important word is "once."
Fallback is not an infinite retry mechanism. It is the minimum safety net that can keep the user flow alive. If that also fails, the system should admit failure. Otherwise, error rate and root cause become harder to observe.
Fallback events are also recorded in metadata. A successful generation does not mean everything was normal. If fallback made it succeed, operators need to be able to see that later.
Why Not Just Retry?
For a temporary outage, retrying the same model can be reasonable. A short retry has value.
But latency matters in an AI generation product. The user is already waiting for the page to be created. Retrying the same overloaded model several times increases the wait. In this product flow, a limited retry followed by a different provider or runtime fit better.
The fallback model still needs to be understood as a different path. Its quality and style may not perfectly match the primary model. So fallback is not "generate anything and mark it successful." It is an alternate path that is better than failing the user flow outright.
Lesson
AI provider failure is not an exceptional event. It is an operating condition.
Products that call model APIs will eventually encounter rate limits, overload, and temporary unavailability. The important work is not eliminating failure. It is classifying failure.
- Is this a failure the user flow can absorb?
- Is this a failure operators should see?
- Is this a configuration issue that needs immediate correction?
- Is this external state that may improve with retry?
If fallback is added without that classification, it hides incidents. If there is no fallback at all, a brief external provider wobble becomes a direct user failure.
How I Would Judge This Next Time
When I see a similar provider failure again, I will not start by adding fallback. I will first separate temporary capacity issues from bad requests, authentication errors, and code defects. The former can be absorbed inside the user flow. The latter should surface quickly.
For future provider fallback work, I will watch three things together.
First, fallback rate should be a metric. If fallback happens too often, the primary provider choice, quota, or model routing needs another look.
Second, fallback output quality should be reviewed separately. Success alone is not enough. The quality gap between primary and fallback output matters.
Third, user-facing messages should distinguish the cases. Users do not need to be told every time a temporary alternate path was used, but when generation fully fails, they should know whether retrying is likely to help.
Reliability in an AI product does not come from trusting one model. It comes from deciding in advance where the flow goes when that model fails.
