The reranker that looked like BERT

I ported a reranker onto the Apple Neural Engine, and the hardest bug in the project turned out to be one line in a config file rather than anything about the hardware.

The model is BAAI/bge-reranker-base, a 278M-parameter cross-encoder with BERT-shaped guts: 12 layers, 768 hidden, GELU. So I reached for the BERT playbook: WordPiece-style assumptions, [CLS]/[SEP], and a single separator between the query and the document. It ran fine, but the relevance scores were quietly wrong.

One line changes everything downstream

The tell was in config.json: model_type: xlm-roberta. If you pattern-match on the architecture you will never see it, because that one line rewires everything the model expects at the input:

• the tokenizer is SentencePiece-Unigram, not WordPiece
• the special tokens are <s> / </s> / <pad> (ids 0/2/1), not [CLS] / [SEP]
• the paired input uses a doubled separator: <s> query </s></s> document </s>

Missing the doubled </s></s> doesn't break anything. There is no crash and no warning; the model just scores against malformed input.

The part I didn't expect was how invisible this is to normal testing. A "does it run" check passes, and the scores even look plausible in isolation. The test that catches it is numeric equivalence against the reference implementation. Two models can share a shape and still disagree on tokenization, separators, and position-id math, and those disagreements never show up as errors, only as wrong numbers.

The lesson travels well past Apple Silicon: read the config, not the architecture diagram. model_type is the real contract.

Once the input was right, the port held

Getting the model resident on the Neural Engine took more than a conversion flag. A naive Core ML conversion of a BERT-family encoder lands on CPU/GPU. To keep it on the ANE you rewrite the graph the way Apple's ane_transformers reference does: every Linear becomes a 1×1 Conv2d, tensors carry a (B, C, 1, S) layout, and LayerNorm is swapped for an ANE-friendly version.

Then you verify it stayed there. My build gate asserts an exact CPU-dispatch fingerprint (the 31 ops that physically can't run on the ANE, like the gather over a 250k-token vocab, casts, and mask math) and zero GPU fallback. If anything drifts off that fingerprint, the build fails.

The payoff, on my machine:

• 2.62ms per pair p95 on the ANE vs 6.92ms on CPU+GPU at batch=20/seq=128, about 2.6x faster
• at seq=256 it's 6.54ms vs 12.16ms, about 1.9x
• FP16 Core ML, a precision conversion, not quantization
• an MTEB SciDocs regression check showing the FP16 conversion cost +0.0005 nDCG@10 vs FP32, which is nothing

Two caveats, because numbers without them are marketing: this is a single-machine benchmark (50 warmup + 100 timed iterations per cell), not independently reproduced. And this is a port of an existing model, not a new reranker; the work was the graph rewrite, the residency gate, and getting the input format right.

The conversion code is at github.com/tcashel/juice-bge-reranker-coreml, and the converted model is on Hugging Face as tcashel/bge-reranker-base-coreml if you're doing on-device inference on Apple Silicon.