PromptLayer Implementation for Prompt Management
When the prompt is the product.
PromptLayer is where the prompts behind an AI product live — versioned, reviewed, tested against real examples and traceable from a bad answer back to the exact version that produced it.
We build on PromptLayer in production for Ontos Admissions Insight, where a twenty-five-year college counseling methodology is held as versioned, reviewable prompts.
PromptLayer is version control, review and evaluation for the prompts behind an AI product. In most AI products the prompt is the part that decides how the thing behaves, and it is the part nobody can see. It sits in a string in a repository, edited by whoever shipped last, with no record of what it used to say, no test that proves the change was an improvement and no route for the person who actually holds the expertise to touch it. That arrangement survives a demo and fails a product. Prompts change far more often than the code around them, the people qualified to write them are usually not engineers, and their regressions are invisible in a way code regressions are not — nothing throws, nothing goes red, the output still reads well and is quietly worse. Treating prompts as production infrastructure is the correction: version history and rollback, a review step before a change reaches a user, an evaluation set that can tell an improvement from a different answer and a trace from what a user saw back to the version that said it.
Our Work with PromptLayer
How the Work Splits
A prompt registry with immutable version history, diffs, release labels and rollback; a visual editor a non-engineer can work in; dataset-backed evaluations with human review and automated graders; production traces tied to the prompt version that produced them, alongside cost, latency and token usage; gradual release and A/B comparison between versions; tracing across agent runs and tool calls; SOC 2 Type 2, HIPAA and GDPR compliance for the teams that have to ask.
The judgment a registry cannot supply. Which part of a methodology belongs in a prompt and which part belongs in the product; the evaluation set that decides whether a change is better rather than merely different; regression tests on the behaviors that may never drift; a review path that puts the domain expert in the authoring seat without putting production in their hands; the integration engineering that ties prompt versions to the application, the model provider and the record of what ran.
AI behavior that can be inspected, argued about and improved by the people who own it. On Ontos Admissions Insight the counseling methodology is encoded as versioned, observable prompts with review, regression testing and traceability — including the rule the whole product rests on, that the AI asks the next question and never writes the student’s essay.
The work in practice
A prompt is the highest-leverage text in an AI product and usually the least governed. Everything around it has version control, code review, a test suite and a deployment record. The string that decides what the model actually does has an author, a date nobody wrote down, and a history that ends at whatever is in the file today.
What the registry is actually for
A history you can read backwards. The useful question about a prompt is almost never “what does it say”. It is “when did it start saying that, and what did it say before”. Immutable versions with diffs answer it; a string in a repository answers it only if someone happened to commit the change on its own.
Rollback that does not need a deploy. A prompt regression should be a release label pointed back at the last good version, not a hotfix branch. Separating the prompt’s release from the application’s release is most of the operational value here, and it is the part teams feel first.
One place the behavior is defined. Prompts multiply — a system prompt, a summarizer, a grader, six tool descriptions — and once they are scattered across three services nobody can say what the product does without reading all of them.
Evaluation is the part that gets skipped
Prompt changes fail silently. The response still arrives, still reads fluently and is worse in a way that only shows up across a hundred examples. So the test cannot be “does it work” tried twice by the person who made the change.
What earns its keep is a dataset of real cases with known-good behavior, run against the new version before anyone sees it, mixing automated graders for what a machine can judge with human review for what it cannot. The set is built out of the failures the product has already had — which makes writing it the cheapest work available, because those failures are already documented in the traces.
The behaviors that must never drift get their own regression tests and their own bar: not “usually correct” but “correct every time, and the build says so”.
The domain expert is the author
The person who knows what a good answer looks like is rarely the person who can open a pull request. When the prompt lives in code, that gap gets closed by an engineer transcribing someone else’s judgment, badly, on a two-week loop.
A visual editor over a versioned registry closes it the other way: the expert writes and tests, the change goes through review and the release label is what promotes it. The discipline that has to come with it is ours to install — who may edit, what has to pass before a version ships, and which prompts are locked because the product’s guarantees depend on them.
Ontos Admissions Insight, in practice
Ontos has spent twenty-five years building a college counseling methodology. Admissions Insight is that method as software, and the method is largely held in prompts: how the AI weighs what a student has already shared, what might be missing and which question would help them go deeper.
That makes prompt infrastructure a product requirement rather than a convenience. The methodology is encoded as versioned, observable prompts with review, regression testing and traceability, so the behavior can be inspected and improved rather than rediscovered. The constraint the product exists to hold — the AI draws the story out and never writes the essay — is built into the product and tested as part of the system, not left to an instruction and a hope.
The users include minors sharing family experiences and setbacks they may not have written down anywhere else, which sets the bar for the record of what the model was asked and what it said.
If you are standing this up
Three things are worth asking about your AI product before anyone compares tools: what the prompt said last month, who approved the change and how you know the new version is better. Where those have answers already, a registry mostly makes them faster to get. Where they do not, the product is running on text nobody owns, and installing the tool is the smaller half of fixing that.
Practice leadership
Christopher Murray
Founder & CEOChristopher Murray founded the firm in 2007 and leads its Applied AI discipline, where he is accountable for how the firm’s AI work is built and for defending it to the people who have to live with it. Nineteen years of delivery in healthcare, higher education and government stand behind the position he holds — that a prompt deciding a product’s behavior is production infrastructure, and belongs under the same version control, review and test as the code around it. He writes for the firm on where AI changes how work actually gets built, and holds the client relationships on the engagements where it does.