Why we don't run open models

The argument we are not going to make

There is a comfortable version of this essay. It says open-weight models are not good enough, that serious products need frontier APIs, and that this is simply what quality costs.

We are not making that argument, because we do not believe it survives the evidence. Stanford’s 2026 AI Index puts the gap between the top US model and the top Chinese model at 2.7%. Whatever the frontier premium buys, it is no longer a category difference, and anyone telling you open weights cannot do the job is describing an earlier era.

There is a second comfortable argument we also have to retire, and this one was ours. The internal shorthand went: to use open models you have to tune them, and tuning needs user data — so open models mean training on you. It is a tidy line. It is also not fully true, and we would rather correct it in public than keep saying it.

Adaptation does not require a data firehose. LIMA aligned a 65B model on a thousand curated examples and got strong results — a demonstration that alignment is largely learned in pretraining and that tuning can be a small, careful act rather than a harvest. And for knowledge specifically, retrieval beats unsupervised fine-tuning: one direct comparison found retrieval-augmented generation consistently outperformed unsupervised fine-tuning for injecting new facts into a model. That happens to be our architecture already. What your twin knows about you, it retrieves. It is not baked into weights.

So the honest position is not that we cannot run open models. It is that we have looked closely at what running them would require of us, and declined.

What self-hosting actually transfers

Choosing open weights is not choosing a different model. It is accepting a different job description.

The lab that serves a frontier model over an API absorbs work that does not disappear when you take the weights: continuous evaluation, safety tuning, red-teaming, abuse monitoring, serving reliability, and the regulatory posture that comes with putting a model in front of the public. Self-hosting moves all of it onto us — every one a discipline we would be starting from zero, and each a place where being merely competent is indistinguishable from being negligent until the day it isn’t.

We already argued, in the self-hosting silo, that running your own agent is a second job. Running your own model is the same trade at a larger scale, with the added property that the failures are less visible. A misconfigured server goes down and you notice. A subtly degraded safety posture does not go down. It just serves.

The asset that would justify it

Here is where the honest argument lands, and it is uncomfortable enough that we understand the temptation to reach for the easy version instead.

There is exactly one asset we hold that would make the self-hosting trade pay: our users’ data. A model tuned on real conversations, real preferences, real business context would be genuinely differentiated in a way no amount of prompt engineering replicates. That is the prize. It is the reason a lot of companies quietly walk this path.

We are not taking that trade, and the reasons are not sentimental.

Tuning on user data leaks it. Fine-tuning with repeated sensitive data raises extraction rates from a baseline of 0–5% to 60–75% — a 64.2% average increase across the models tested. That is not a theoretical attack surface; it is a measured property of the technique. Mitigations exist and they work: the same paper’s protections cut leakage to zero while keeping 94.7% of utility, and differentially private training and low-rank adaptation both reduce exposure. They reduce it. Whether any of them eliminates it for a specific corpus is a question you answer with an audit you have to be qualified to run, on a risk you chose to create.

Trained weights are not automatically anonymous. The European Data Protection Board’s Opinion 28/2024 addressed this directly: a model trained on personal data cannot simply be declared anonymous by virtue of being a model. Whether it is depends on how hard it would be to extract that data back out. The weights inherit the obligations of what went into them.

The remedy runs to the model itself. The FTC’s action in the Everalbum matter required deletion not only of improperly obtained photos but of models and algorithms derived from them. Algorithmic disgorgement means a data-provenance mistake does not cost you a dataset. It costs you the thing you built with it, after you have built your product on top of that.

The regulatory category changes. Article 53 of the EU AI Act attaches a specific set of duties to providers of general-purpose AI models — technical documentation, downstream disclosure, copyright policy, training-data summaries, cooperation with authorities. Open-source releases get a carve-out from some of it, but not where systemic risk is in play. Today those obligations sit with the labs whose models we call. Placing our own model on the market is a decision to move some of that weight onto ourselves, and it should be made deliberately or not at all.

Tuning erodes alignment even when you mean well. Qi et al. showed that fine-tuning degrades a model’s safety alignment even with entirely benign data — not as an attack, as a side effect. The safety properties we would be relying on are the ones our own tuning would quietly spend.

And synthetic data has a ceiling. The obvious escape — tune on generated data instead of user data — runs into model collapse: train recursively on generated content and the distribution’s tails disappear, degrading the model toward its own average. Synthetic data is a useful supplement and a poor foundation.

The trade, stated plainly

We do not build models. Inference runs on frontier labs’ APIs, and what your twin learns, it learns for you and keeps for you.

That choice costs us something real. We are dependent on providers we do not control, exposed to their pricing and their deprecations, and unable to fine-tune our way to a moat. We accept those costs with our eyes open, and manage the dependency rather than pretending it away.

What we get in exchange is that there is no version of our roadmap where your conversations become our weights — not because we promised, but because the architecture never creates the artifact. A promise can be revised by a future executive under pressure. An absent model cannot leak.

The open-weight ecosystem is doing important work and the quality gap has all but closed. Our objection was never to the models. It was to what we would have to become to run them.