CTO simulator / AI product

Ship an AI product.
Five calls, one feature.

These are the calls I see teams make when they ship an AI feature: which model, how much context, how much to test, where the data goes, and how fast to roll out. I built this simplified model to think through how they pull against each other.

Decision studio

Five calls, one feature.

Make each call, watch the indexes move, then read the debrief. It follows simple rules, and there is no single right answer.

Choices update the URL. Nothing is stored.

Model strategy

Where should capability, speed, and spend go?

Context and memory

How much about the user and their history should the model see?

Evaluation

What evidence do you want before and after launch?

Data and privacy

What boundary should surround the user data the model sees?

Rollout

How quickly should real users see it?

CTO readout

The debrief

Workable, with clear trade-offs.

What’s working

  • Routing saves the expensive model for the requests that need it instead of paying for it on every call.
  • Targeted evals spend the effort on the flows most likely to fail, without testing everything.
  • Sending only the fields you need, with identifiers stripped, limits what can leak.

What could break

  • Routing, context, and evals still need a cost ceiling per request as usage grows.

Continue the work

Go deeper.

Talk through this plan in chat

AI evals and measurement

Turn quality goals into test sets, rubrics, and release gates.

Inference economics

What drives latency and cost per request, and where routing helps.

Securing agents

Data boundaries, permissions, and prompt injection.

AI agents in production

Getting from a demo to something you can watch and roll back.