Finish One Fine-Tune in 33 Seconds: LoRA on Apple Silicon

Thirty-three seconds. I hit Enter on my Mac, poured a glass of water, and came back to a model that already knew how to emit the JSON schema I wanted.

Not a cloud GPU. Not a rented A100. Not a "theoretically possible but nobody actually does this" proof of concept. Just a MacBook Pro with an M4 Pro, finishing 60 steps of LoRA training, with loss falling from 1.06 to 0.17 and peak memory under 7GB. The fine-tuned model then spat out clean single-line JSON: no markdown fences, no pretty-printed indentation, just the format I asked for.

This post records the full path: why I chose Gemma-4 E4B, which traps I hit, how I worked around them, and what the results looked like. If you also run local models on a Mac and want them to fit your hand better the more you use them, this path has already been validated.

Why Gemma-4 E4B

The story starts with oMLX benchmarks.

I keep four local models resident and ran a round of performance tests with community oMLX scores. The results were clear:

Model	Prefill (tok/s)	Generation (tok/s)	Memory footprint
gemma-4-e2b 4bit	2,090	133.7	~3.5GB
gemma-4-e4b 4bit	1,117	~90	~6.8GB
Qwen3.6-35B-A3B-UD 4bit	777	69.9	~20GB
qwen3-4b	721	-	~2.5GB

In the same weight class, Gemma-4 E4B is clearly faster at inference than qwen3-4b, and long-context performance (up to 64K) degrades gently. More important for me: it only uses about 6.8GB, so LoRA training on a 48GB machine has plenty of headroom.

But there was one thing I did not notice at first.

You Think It Is a Text Model. It Is a VLM

Open Gemma-4 E4B's config.json and you suddenly see ForConditionalGeneration, vision_config, and audio_config. This is not a pure text language model. It is an omni multimodal model with a vision tower, an audio tower, and a text tower at the same time.

More surprising still: my main model, Qwen3.6-35B-A3B, is also a VLM. Of the four local models I run, not a single one is a pure text causal LM.

What does that mean? It means you cannot take the pure-text fine-tuning path of mlx_lm.lora. Because these are VLMs, you have to load through the mlx-vlm flow. My task, though, is pure text (NER entity extraction), so the strategy is simple: freeze the vision tower and audio tower, and train only the language layers inside the text tower.

The Trap: KV-sharing Regression in mlx 0.31.2

The environment was ready. Python 3.12 virtualenv, mlx-tune 0.6.0, mlx 0.31.2, mlx-vlm 0.6.3, all latest from PyPI. I downloaded the lmstudio-community gemma-4-E4B-it-MLX-4bit (flat layout, 6.8GB) and prepared to train.

It blew up on load:

ValueError: Received 126 parameters not in model:
  layers.*.self_attn.k_norm
  layers.*.self_attn.k_proj
  layers.*.self_attn.v_proj

Try e2b instead? Same error, only the count becomes 140. The whole Gemma-4 e series is wiped out in the current mlx ecosystem.

Root Cause

After digging, this turned out to be a regression introduced when mlx moved from 0.31.1 to 0.31.2 (mlx-lm issue #1242, opened May 2026, still unfixed at the time of writing).

Gemma-4's architecture uses a KV-sharing mechanism: some attention layers share K/V projections, and quantized export produces a batch of "round-trip redundant copies." Those tensors are unused at inference time, and the model definition has no matching parameter names for them. mlx 0.31.1 silently ignored them. Starting with 0.31.2, the check is strict, and strict=True rejects them outright.

Why the "Official Fixes" All Fail

First instinct: downgrade. But both mlx-vlm and mlx-lm pin mlx>=0.31.2, and PyPI has already removed 0.31.1. The dependency chain blocks that path.

Second instinct: use strict=False. The lower-level nn.Module.load_weights() in mlx does expose that argument. But when mlx-vlm calls load_weights internally, it does not forward strict=False, so anything you set at the outer layer has no effect.

Fix: Monkeypatch

If the public API cannot get you there, patch the bottom layer. At the very top of the training script, monkeypatch mlx.nn.Module.load_weights so strict is forced to False:

import mlx.nn as nn
_orig_load = nn.Module.load_weights

def _patched_load(self, file_or_weights, strict=False):
    return _orig_load(self, file_or_weights, strict=False)

nn.Module.load_weights = _patched_load

Those redundant tensors are then silently dropped, and the model loads as usual. In practice, generation produced coherent Traditional Chinese, so the architecture is fully correct. The logic of this workaround is simple: those K/V tensors were redundant copies to begin with, and dropping them does not hurt inference.

Training: 45 Samples, 60 Steps, 33 Seconds

Once loading worked, training itself was the easy part.

My task is six-dimension NER (named entity recognition): given a biomedical passage, the model must output a JSON object covering six dimensions. Training data was generated by script: 45 train samples and 5 valid samples in ChatML jsonl format.

LoRA hyperparameters:

Parameter	Value
rank (r)	16
alpha	16
dropout	0
target modules	q/k/v/o/gate/up/down_proj
batch size	1
gradient accumulation	4
learning rate	2e-4
max steps	60
max sequence length	512
train_on_completions	True

Sixty steps finished in about 33 seconds. Loss fell from 1.06 all the way to an average of 0.17. Peak memory was about 6.9GB, far below the 48GB ceiling, so other apps were never affected.

Before vs After: The Power of Format Compliance

The effect of fine-tuning is obvious in a before/after comparison.

Before (base model, no fine-tuning):

After receiving the NER instruction, the model did identify the right entities, but the output format was completely non-compliant:

• Wrapped in a triple-backtick json fence
• Multi-line pretty-printed indentation
• Case drift (lowercase fields written in uppercase)
• Extra post-processing required before the output could feed a downstream pipeline

After (with LoRA adapter attached):

• Emits compact single-line JSON directly
• No fence, no extra whitespace
• Correct casing
• Of 4 validation sentences, 3 matched the gold standard exactly
• 1 case showed HER2 protein/gene category confusion, a small content-level error that more training data can fix

The point is: fine-tuning hits "format compliance" and "style fit" almost immediately. You do not need thousands of samples or hours of training. With 45 examples and 33 seconds, the model already learned the output structure you want.

Content accuracy is a data quality and quantity problem, not a limit of the fine-tuning mechanism itself.

The "Fits Better the More You Use It" Loop

The core claim of this MVP is not "can we fine-tune," but whether the loop of daily use, periodic retrain, and adapter swap actually holds.

It does. And the cost is surprisingly low.

The loop looks like this:

1. Daily use: run local models on everyday tasks (NER, classification, summarization, format conversion)
2. Collect samples: mark good model outputs and accumulate them as training data
3. Periodic retrain: run LoRA weekly or monthly, tens of seconds to a few minutes
4. Swap adapter: ship the new adapter so model behavior tracks your needs more closely
5. Return to step one

This is not online learning or live weight updates. Those approaches are unstable on large language models, so nobody runs them in production. In practice you batch retrain and hot-swap adapters: simple, controllable, and reversible.

A few things to keep in mind:

• Fine-tuning learns "fit" (format, voice, domain terms), not "higher IQ." The model's capability ceiling does not rise because of fine-tuning.
• Bad data can cause catastrophic forgetting. You need quality review and evaluation gates.
• Leave factual knowledge to RAG and the memory layer. Fine-tuning only owns style and skill.

Side by Side with T15: Mac vs Windows

If you read T15 (QLoRA fine-tuning on Windows), the trade-offs of the two paths look very different:

Dimension	T15 Windows QLoRA	T16 Mac LoRA
Hardware	NVIDIA GPU (RTX 3060 or better)	Apple Silicon (M1 or later)
Stack	Unsloth + PyTorch + CUDA	mlx-tune + MLX
Ecosystem maturity	Mainstream, lots of tutorials	Fringe, you walk the traps yourself
Fine-tune speed	Depends on GPU, usually faster	33 seconds / 60 steps (E4B)
Memory model	Separate VRAM, does not touch the system	Unified memory, watch the total
Biggest pain	Driver and CUDA version conflicts	mlx ecosystem regressions and compatibility

Both roads reach the same end state of "fits better the more you use it." The difference is that the Windows/NVIDIA side is a paved highway with heavy traffic and clear navigation. The Mac/MLX side is a mountain road where you sometimes get out and move rocks, but the view is good, and you do not need to buy a separate GPU.

What Comes Next

This MVP validated the minimum loop. Next up:

• Grow the training set: fix the HER2 protein/gene disambiguation issue, and add more biomedical NER examples
• Plug into a daily collection pipeline: automatically gather high-quality prompt-response pairs from daily use as material for the next retrain
• Dual-model routing: the fine-tuned E4B handles high-throughput pre-processing (NER, triage, labels), while the 35B Qwen keeps carrying the heavy reasoning work
• Try more tasks: beyond NER, Traditional Chinese science popularization first drafts, IRB academic voice, and other workloads wait to be tested

Thirty-three seconds can do more than you think. No GPU rental, no cloud queue, no uploading data anywhere. Your Mac is your training box, and your daily use is your training data. The model will fit better the more you use it, as long as you are willing to spend 33 seconds teaching it once.