---
title: "Perplexity's Lily Engine: Running a 35-Billion-Parameter Model on a Laptop"
slug: perplexity-lily-on-device-llm-consumer-hardware
category: tech
category_label: "Tech"
author: "BrainWavePost Staff"
date: 2026-09-04
tags: ["AI", "Perplexity", "Edge AI", "On-Device Inference", "Lily", "Qwen", "Apple Silicon", "Quantization"]
canonical_url: https://brainwavepost.com/article/perplexity-lily-on-device-llm-consumer-hardware
source: BrainWavePost
---

# Perplexity's Lily Engine: Running a 35-Billion-Parameter Model on a Laptop

*Tech · 2026-09-04 · BrainWavePost Staff*

> Perplexity open-sourced Lily, a Rust + Metal inference engine that runs a 35-billion-parameter Qwen model on Apple Silicon — not by upgrading the chip, but by rewriting the software that sits between the model and the silicon.

A 35-billion-parameter AI model is not something most people expect to run on a laptop. Yet in September 2026, AI startup Perplexity demonstrated exactly that — running Alibaba's Qwen3.6-35B-A3B model on a MacBook and then making it faster, not by using better hardware, but by rewriting the software that maps the model onto the chip. [1][2]

The engine behind that work is called Lily, and on September 2, 2026, Perplexity's CEO Aravind Srinivas announced the company was open-sourcing it. Lily is the local half of Hybrid Compute, a split-execution mode in Perplexity's Mac app that divides work between on-device processing and Perplexity's cloud — a design aimed at letting users run capable models locally without paying cloud token costs or sending private data off the device. [1][2][4]

## 1. What Perplexity actually shipped

Lily is not a general-purpose inference server. Perplexity built it specifically for one model — Qwen3.6-35B-A3B — and one hardware family, Apple Silicon, with separate hand-tuned optimizations for the two stages of inference: prefill (reading the prompt) and decode (generating the reply). [1][2]

The architecture runs as a single process with three components working together: a Rust runtime that loads the model checkpoint and drives the generation loop; an OpenAI-compatible chat-completions API that lets existing tools talk to Lily with minimal changes; and custom Metal kernels, written by hand rather than generated by a general compiler, that execute the Qwen-specific tensor operations directly on the Apple GPU. Neither PyTorch nor Apple's own MLX framework sits anywhere in the execution path. [2]

> **Why this is unusual** _(info)_
>
> Most local-AI tools — llama.cpp, Ollama, LM Studio, Apple's MLX-LM — are general-purpose: they support many models across many chips. Lily throws that flexibility away for speed. Because Perplexity knows in advance exactly which tensor shapes and memory layouts Qwen3.6-35B-A3B will produce, its engineers could write Metal kernels tuned to that one case rather than kernels that must handle arbitrary models reasonably well. [2]

## 2. The model: why a 35B model fits on a laptop at all

Qwen3.6-35B-A3B comes from Alibaba's Qwen team. The name encodes the shape: 35 billion total parameters, built as a mixture-of-experts (MoE) architecture. In an MoE design, the full parameter count is not active for every token — the model routes each token through a small, relevant subset of its 'expert' sub-networks, which makes it cheaper to run than its headline size suggests. [1][2]

The model also uses two different kinds of internal layers. Some use full attention, which looks back at everything said so far. Others use a newer method called Gated DeltaNet, which keeps a running summary of past context instead of storing everything and is generally faster and lighter on memory. Handling both layer types adds complexity that a general-purpose engine finds harder to optimize — and that is exactly the gap Perplexity targeted. [1]

## 3. The three optimizations that mattered

### 4-bit quantization: 70 GB down to ~19.4 GB

The first step was quantization — shrinking the model the way a large photograph can be compressed into a smaller file. Perplexity compressed the model from roughly 70 GB to about 19.4 GB using 4-bit quantization, making it small enough to fit in a laptop's unified memory. The technique is not new: PrismML had used similar quantization to compress Alibaba's Qwen3.6 27B model for consumer hardware, and Apple's own machine-learning team ran Llama 3.1 8B on an M1 Max at about 33 tokens per second in 2024 using 4-bit quantization plus a stateful cache. [1]

### On-the-fly decompression inside the chip

The harder problem is that a compressed model still needs to be decompressed before the chip can calculate with it, and that decompression step usually writes the expanded numbers back into memory first — creating extra work. Perplexity's solution decompresses each small piece at the moment it is needed, inside the chip itself, without first writing the expanded version to memory. The company said this change alone made prompt processing up to 77.4% faster in one test. [1]

### Cutting memory movement during decode

Apple's chips use unified memory, where the CPU and GPU share the same pool of memory — which avoids some copy delays but does not remove the problem of a chip waiting for data to arrive. Perplexity reduced how often the model needed to move data in and out of memory during response generation, partly by keeping more calculations on the GPU rather than sending data back to the CPU unnecessarily. None of this changed the underlying chip; what changed was how completely the software used it. [1]

## 4. The benchmark numbers

Perplexity ran its benchmarks on an M5 Max MacBook Pro, testing across ten prompt lengths and ten decode contexts, comparing Lily against Apple's MLX-LM running the identical Qwen3.6-35B-A3B checkpoint. The headline results: Lily averaged 1.23x higher prefill throughput and 1.35x higher decode throughput than MLX-LM. [1][2]

Crucially, the speed gain did not come at the cost of output quality. Independent analysis found Lily's outputs showed roughly 0.04% higher perplexity than MLX-LM's, alongside 96.35% identical top-token selection between the two engines — meaning Lily produces almost the same words in almost the same order, just faster, rather than trading accuracy for speed. [2]

- **1.23x** — Faster prefill (prompt processing) vs Apple's MLX-LM on M5 Max [2]
- **1.35x** — Faster decode (token generation) vs MLX-LM on the same chip [2]
- **~19.4 GB** — Model size after 4-bit quantization, down from ~70 GB [1]
- **96.35%** — Top-token agreement with MLX-LM — near-identical output [2]

## 5. Hybrid Compute and the local-first vision

Lily powers the local side of Hybrid Compute, the split-execution mode in Perplexity's Computer app for Mac. The idea is that the entire stack — model, harness, conversation and trajectory — lives on the user's machine by default. Work that needs the outside world, such as web search or escalation to a stronger cloud model, is invoked only when necessary and is gated by the user, so sensitive data never leaves the device without permission. [3][4]

Perplexity frames this as a response to three competing pressures: accuracy wants the most capable (and expensive) models; privacy wants some work to never leave the machine; and cost and energy mean you should not spend a frontier model's compute on a task a smaller one can handle. The stated goal is the most token value per watt, per user. [3]

The same local-first philosophy underpins Perplexity's Portable Computer, launched on August 25, 2026 in partnership with Nvidia — a version of its agentic platform that runs entirely on hardware users already own, starting with Nvidia's DGX Spark desktop and Linux machines with RTX GPUs, with zero token costs. [5][6]

## 6. Why Perplexity gave the code away

Open-sourcing a hand-optimized performance edge is unusual for a well-funded AI company. The practical reason is that if the best local performance requires a custom engine for every model-and-chip combination, running AI locally becomes extremely difficult to maintain — which is why general-purpose frameworks like MLX-LM and llama.cpp exist in the first place, trading some performance for the ability to work across a wide range of hardware. [1][2]

By releasing Lily under an open-source license, Perplexity lets the specific techniques it uses — on-the-fly in-chip decompression, reduced decode-time memory movement, hand-tuned Metal kernels for MoE routing — filter into more general tools over time, rather than staying locked inside one product. [1][2][7]

## 7. The broader on-device trend

Perplexity's work is part of a wider push toward edge AI. Microsoft took a different approach with Phi Silica, a small model built into Windows Copilot+ PCs: because those laptops include a dedicated neural processing unit (NPU) that is more limited than a full GPU but far more power-efficient, Microsoft designed Phi Silica around that constraint using speculative decoding — where a smaller, faster model guesses the next few words and a larger model quickly checks them. [1]

Apple's own 2024 research showed Llama 3.1 8B running on an M1 Max at about 33 tokens per second with 4-bit quantization and a stateful cache. Community benchmarks for Qwen3.6-35B-A3B under llama.cpp show roughly 105 tokens per second at short context and around 97 tokens per second at an 8,000-token context using about 45 GB of unified memory at Q8 — separate community data, not a direct head-to-head against Lily. [1][2]

The shared lesson across all of these cases is the same one Perplexity's experiment highlights: the software layer sitting on top of the chip, deciding how a specific model's calculations map onto that specific hardware, determines how much of the chip's potential is actually usable. As AI moves from the cloud onto laptops and phones, chip specifications will keep improving, but they will tell only part of the story. [1]

## 8. The accuracy box: what is solid and what is not

- Solid: Perplexity open-sourced Lily on September 2, 2026, a Rust + Metal inference engine for Qwen3.6-35B-A3B on Apple Silicon. [1][2][7]
- Solid: Lily averaged 1.23x faster prefill and 1.35x faster decode than MLX-LM on an M5 Max MacBook Pro running the same model. [1][2]
- Solid: 4-bit quantization compressed the model from ~70 GB to ~19.4 GB, and on-the-fly in-chip decompression made prompt processing up to 77.4% faster in one test. [1]
- Careful with the framing: Lily is purpose-built for one model on one chip family — it is not a general-purpose engine, and the gains are specific to Qwen3.6-35B-A3B on Apple Silicon. [1][2]
- Not proven: that this approach scales — Perplexity's own testing showed some optimizations improved individual steps without meaningfully speeding up the full response, and some steps were already near the chip's limits. [1]
- Subject to change: the benchmarks are vendor-published and self-reported; independent head-to-head comparisons against llama.cpp or other runtimes are not yet available. [2]

## 9. What to watch next

1. Whether Lily's specialized techniques — on-the-fly decompression, reduced memory movement — get adopted by general-purpose frameworks like MLX-LM or llama.cpp. [1][2]
2. Whether purpose-built, single-model engines become a practical pattern or remain a costly exception, since maintaining a custom runtime per model-chip pair is hard. [1]
3. How Perplexity's Hybrid Compute split between local and cloud evolves, and whether the local-first privacy promise holds as the product scales. [3][4]
4. How the Nvidia-partnered Portable Computer — fully local on DGX Spark and RTX hardware — compares on real-world agentic workloads versus the cloud-assisted Mac app. [5][6]
5. Whether Apple, Microsoft and other platform holders bake similar on-device optimizations into their own first-party inference stacks, closing the gap with specialized engines like Lily. [1]

## The bottom line

Perplexity's Lily is a real, open-source demonstration that a 35-billion-parameter model can run on consumer Apple Silicon hardware at usable speeds — not because the chip got better, but because the software between the model and the silicon got smarter. [1][2] The deeper signal is that as AI moves to the edge, raw chip specs will matter less than the inference-engine software that decides how to use them — and the tradeoff between specialized speed and generalized portability is now the central engineering question for on-device AI. [1][2][3]

## Sources and further reading

- [1] Business Standard — 'Perplexity may have a way to run LLMs on consumer hardware: How it works' (Harsh Shivam, September 3, 2026): https://www.business-standard.com/technology/tech-news/perplexity-run-llms-consumer-hardware-how-it-works-126090300889_1.html
- [2] Tech Insider — 'Perplexity Open-Sources Lily: 1.35x Faster Than MLX' (Elias Virtanen, September 3, 2026): https://tech-insider.org/perplexity-open-sources-lily-inference-engine-2026/
- [3] Perplexity — 'The Data Center Moves to Your Machine' (blog): https://www.perplexity.ai/hub/blog/the-data-center-moves-to-your-machine
- [4] Perplexity — 'A Local-First Agent for Private Knowledge Work' (blog): https://www.perplexity.ai/hub/blog/a-local-first-agent-for-private-and-cost-effective-knowledge-work
- [5] VentureBeat — 'Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs' (August 25, 2026): https://venturebeat.com/infrastructure/perplexity-partners-with-nvidia-to-launch-portable-computer-a-fully-local-ai-agent-with-zero-token-costs
- [6] The Verge — 'Perplexity's new Portable Computer runs entirely on device' (August 25, 2026): https://www.theverge.com/ai-artificial-intelligence/984368/perplexitys-new-portable-computer-runs-entirely-on-device
- [7] The Crypto Post — 'Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon' (September 2026): https://thecryptopost.io/perplexity-open-sources-lily-a-rust-metal-inference-engine-for-qwen3-6-35b-a3b-on-apple-silicon/

---

_Canonical article: [https://brainwavepost.com/article/perplexity-lily-on-device-llm-consumer-hardware](https://brainwavepost.com/article/perplexity-lily-on-device-llm-consumer-hardware) — © BrainWavePost. Educational content; see the article page for full disclaimers._