Local AI Experiment

More of the intelligence moved into the system.

The Question

Can a completely local AI, running on an ordinary laptop, come to feel like it knows how I work?

Not by training. The model never changes. The bet is that accumulation does the work instead: the system records my decisions and corrections as I use it, retrieves the right ones at the right moment, and places them in front of the model before it answers. If that loop works, the answers should start to feel personal within about four weeks of real use.

One rule frames everything: nothing leaves the hard drive. No external APIs, no cloud inference. For a photographer holding a client's unpublished campaign, or a studio working under NDA, that constraint is not a preference. It is absolute.

Not a Demo, an Experiment

Getting a model to run on a laptop is a demo. This is set up as an experiment: a hypothesis, a fixed way of measuring, and failure conditions declared before starting. If retrieval ever feels like a slower way to search my own files, if corrections never change the answers, or if I simply stop using it, the hypothesis dies. A rejection published honestly is worth as much as a success, and either way the result appears here.

The Floor Test

The Machine

The test machine is deliberately modest: a consumer laptop GPU, an RTX 3060 with 6GB of VRAM, with Ollama as the runtime. If it works comfortably here, the hardware requirement stays within reach of many working creatives. Five open models were pulled onto it and benched under identical conditions.

Speed and Heat

The two columns matter more than either alone. This class of laptop throttles fast and hard: the GPU idles at 63°C, reaches 88 to 96°C within about two prompts, and on consecutive prompts I measured a drop from 46.9 to 15.1 tokens per second. Cold-start numbers flatter every model. The sustained column is the one you actually live with.

The reading above is the whole constraint in one frame: 92 degrees, and 5491MiB of the card's 6144MiB already spoken for. The 8B class held 7 to 9 tokens per second regardless of temperature. That is reading speed, not working speed, and it ruled the tier out on this hardware. The 14B ran slower still. On 6GB of VRAM, the realistic zone is the 4B class.

Gemma Against Qwen

The real contest was between the two 4B models. Gemma 3 4B won on sustained speed, and reading their answers side by side, its Japanese writing was the strongest of the five. Its one gap in this setup: the standard Gemma 3 4B model cannot drive search tools directly. The finding below made that gap irrelevant.

The Finding

Two Conditions, Same Model

I ran the same 20 retrieval tasks, 14 in Japanese and 6 in English, over an identical seven-file project corpus with automated pass and fail scoring, under two conditions.

Qwen 3 4B searching on its own with tools succeeded 40 percent of the time. Handing the same material directly to the model: 90 percent.

Same model.

And Gemma, with no tool use at all in this setup, matched the same 90 percent once the material was supplied.

On the scoring, in full: raw automated scores were 16 of 20 for Qwen and 17 of 20 for Gemma. Two of the twenty checks could only pass by making a tool call, which the no-tools condition cannot do by definition, so those two are not comparable across conditions and are excluded from the analysis. Scored over the 18 eligible tasks, both models reach 90 percent. The excluded checks are named in the bench report.

Prompting alone did not rescue the tool loop in these tests. A search-first prompt made it worse, with the model looping in search until its rounds ran out. Four tool-capable models were measured the same way, from 4B up to 14B, and none came close to the bar I had set. For this task set and tool setup, model-led retrieval was not reliable enough to use.

One more result mattered to me as a bilingual user. Cross-language retrieval also worked in the cases I tested: a question asked in Japanese could retrieve an answer written in English.

The Architecture Decision

The bench retired an architecture, which is exactly what a bench is for. In this setup, letting the model own retrieval was not reliable enough, and prompting alone did not fix it. So the application owns retrieval instead, along with memory, routing and guardrails, and the model does the one thing it is genuinely good at: reasoning over supplied context in the user's language.

The memory itself is ordinary files on the machine. The model keeps nothing. It only visits.

Memory

How It Learns

Learning happens entirely outside the model. When I correct something, the correction becomes a candidate card, and nothing is written until I approve it. Approved cards are plain files I can read, edit or delete, and the next answer draws on them.

Below is the first time that loop closed. A preference stated once, in an earlier session, comes back as the basis of an answer, and the memory command shows exactly which card was used.

The exchange runs over a rehearsal project rather than a real client, which is the point of having one: the loop can be tested against invented work before it ever touches material that matters.

Living With It

The App

The command line proved the loop. Living with it needed a surface. The experiment now runs as a small local app on the same machine: a chat surface, a memory pane where everything it has learned stays visible and editable, and the approval gate carried over from the prototype.

What I Ask It

There is no synthetic test set for this phase. The measure is real use, so the questions come from a working bank tagged by what each one actually tests.

Status

• 私の残ってるタスクは？ What tasks are still open?

• 今日は2時間しかない。何をすべき？ I only have two hours today. What should I do?

Retrieval

• MokuDocs 2.0の価格いくらにしたんだっけ？ What price did I settle on for MokuDocs 2.0?

• 香川のシュートで何かトラブルなかったっけ？ Was there trouble on the Kagawa shoot?

• 今年の収益目標いくら？ What is this year's revenue target?

Drafting

• 納期を1週間延ばしてほしいと伝えるメールを下書きして Draft the mail asking to move a deadline by a week.

• Instagramのキャプション案を1つ、私のスタイルで One Instagram caption, in my style.

Judgment

• セレクトで縦と横で迷ってる。どっちにすべき？ Torn between the vertical and the horizontal frame. Which one?

• LinkedInの次の投稿ネタどうする？ What should the next post be about?

Teaching

• 金曜は撮影優先の日 Friday is for shooting.

• 迷ったら出荷を優先する When in doubt, ship.

The working bank holds around forty prompts across these five categories. Status and retrieval are no longer the main unknowns in this experiment: the bench established a workable approach for both. The last three tags are where it will be decided. A drafting request has to come back as the actual text. A judgment call is only useful if the answer reflects how I actually work. And a preference stated once has to survive.

Verdict Pending

Much of what users experience as intelligence is not the model. It emerges from the system designed around it. That is the working conclusion so far, and the destination is to build it into the desktop app I am developing, where knowledge accumulates from everyday work and everything stays on the machine.

The experiment runs a few more weeks before that call is made. Either outcome is a result, and both will be published here.

View the full experiment in the Lab →