At first, Place2Page prompts looked like ordinary strings. As experiments accumulated, the real problem became less about finding a better sentence and more about whether we could later explain which candidate won, on which input, and why. So temporary prompt changes stopped being one-off request strings and became part of an operational record: experiment group, candidate, judgment, snapshot, and production promotion.
Summary
- Problem: Copy-paste prompt experiments made it difficult to trace which wording produced which result.
- Decision: Prompt changes needed to be recorded as baselines, candidates, judgments, and promotion decisions, not only as "looks better."
- Fix: Record experiment groups, candidates, judgments, prompt snapshots, and production promotion in one workflow.
- Lesson: Prompts become operable assets when they have experiment, record, comparison, and promotion workflows, much like code.
Context
Place2Page generates landing pages from place data. A user enters a Google Maps or Naver Map URL, and the service uses place information such as photos, opening hours, menus, and reviews to produce an HTML page.
In the beginning, editing prompts was simple. Change one sentence, inspect a few generations, and decide whether the output looked better. After a while, that stopped scaling.
It became hard to remember which prompt produced which result. Candidate A seemed good, but the exact wording was difficult to find again. There had also been a slightly modified B candidate, and it was unclear whether it had only been tested or actually promoted to production.
As prompt experiments increased, copy-paste and memory became obvious weak points.
Problem
The riskiest state in prompt experimentation is stopping at "this looks good."
Even if one prompt appears better, it is hard to operate unless the team can answer these questions:
- Which place input was used?
- What was the baseline, and what changed in the candidate?
- Why was it judged better?
- Which cases failed?
- Can the same prompt be selected again later?
- Is this prompt the current production default?
Prompts are easy to change because they are strings. That also makes them easy to scatter. If experiment results do not leave a code-diff-like trail, finding "that wording from last time" becomes a debugging task.
Initial Hypothesis
I first thought moving prompt text into files would be enough. Files leave git diffs and can be reviewed.
But real experiments did not end at files. I needed to rerun a specific place, apply a temporary prompt change, or compare multiple candidates against the same input. The experiment unit was not one file. It was "which prompt was applied to which input, and how was the result evaluated?"
That meant prompt file management needed a layer above it: experiment management.
Evidence
This was not only a vague concern about future maintainability. Repeated experiments made the needed record shape clear.
A single problem is grouped into one experiment, and baseline and candidate prompts are compared inside it. After inspecting generated output, the result is recorded as passed, held, or needing rework.
The more important evidence is in tests. Even when a temporary prompt change is applied, the raw string is not allowed to remain only inside the request payload. The test verifies that the changed prompt is saved as a snapshot and that generation refers to that saved snapshot instead of carrying the raw string forward.
Experiment: copy quality check
Candidate: stricter grounding rule
Saved prompt snapshot exists
Generation request references the saved snapshotSo experiment wording does not disappear as a one-request string. It becomes a record that can be selected and compared again.
The same structure mattered later when checking the production default. The default prompt in files and the prompt actually selected at runtime can differ. Without that check, it would have been hard to explain why production output changed while the file stayed the same.
Implementation
The workflow makes the experiment unit explicit.
- Experiment group: one unit for validating a problem
- Candidate: the prompt or setting being compared inside that problem
- Judgment: the current decision on the result
- Notes: a short record of why that judgment was made
For example, if duplicate external map links appear even though an embedded map is present, that issue becomes one experiment group. Inside it, a baseline and a stricter candidate can be compared. Results can then be labeled as passed, needing rework, or held.
The key piece is the prompt snapshot.
When a temporary prompt is applied, the raw string is not stored only in the request payload. The baseline prompt is copied, and the changed part is saved as a new snapshot. The generation request then refers to the saved snapshot instead of the raw string.
That makes the experiment prompt recoverable. It can be opened again, edited, applied to another input, or promoted to production if it proves useful.
Why Production Promotion Was Needed
Prompt experiments need a separate path for the moment when "this seems best" becomes the actual default.
If the system only edits a file or environment variable, it becomes hard to verify what production is currently using. If a verified prompt snapshot is promoted to production, the next generation uses that snapshot as the runtime default, while the file default remains as a fallback.
The benefits are concrete:
- File default prompts and runtime production prompts can be distinguished.
- A verified experiment snapshot can be promoted directly to production.
- If a problem appears later, the promoted change can be traced.
- The current runtime default can be checked separately.
If prompts are deployed like code, their deployment state should also be inspectable like code.
Lesson
This work changed how I think about prompt engineering.
At first, I thought the main job was finding better prompt wording. The wording still matters. But inside a product, the more important question is whether the prompt can be experimented with, selected by criteria, and traced once it becomes production.
Prompt experiments disappear easily if they live in notes. When they are managed through experiment groups, candidates, judgments, snapshots, and promotion, they become operational assets.
In Place2Page, changing a prompt is no longer described as just "edited the prompt." The record also needs to say which problem we saw, which hypothesis we tested, which candidate won, and why.
How I Would Judge This Next Time
For future prompt experiments, I will first ask whether the decision can be reproduced later, not whether the wording feels good. An unrecorded prompt win is closer to memory than an operational asset.
At minimum, a prompt experiment should preserve:
- Problem context
- Experiment input
- Baseline
- Candidate
- Judgment
- Failed cases
- Production promotion status
Prompts are easy to change, so they need stricter records. Copy-paste can start an experiment, but it is a poor long-term operating model.
