Google Releases LiteRT.js: A JavaScript Binding of LiteRT That Runs .tflite Fashions in Browsers through WebGPU

Google launched LiteRT.js, a JavaScript binding of LiteRT. LiteRT is Google’s on-device inference library, beforehand referred to as TensorFlow Lite.

LiteRT.js runs .tflite fashions instantly contained in the browser. As a result of inference stays native, Google cites enhanced person privateness, zero server prices, and ultra-low latency.

What’s LiteRT.js?

It’s not a brand new mannequin format. Quite, Google compiled its current native runtime to WebAssembly and uncovered it to JavaScript.

Earlier internet AI options, together with TensorFlow.js, relied on JavaScript-based kernels. Google describes these as much less performant. LiteRT.js as an alternative ships the native cross-platform runtime with its optimizations intact.

Consequently, internet apps inherit work finished elsewhere. Efficiency upgrades, quantization enhancements, and {hardware} optimizations constructed for Android, iOS, and desktop arrive on the net too.

How It Works: One Runtime, Three Backends

Underneath that runtime, LiteRT.js targets three backends:

• CPU makes use of XNNPACK, Google’s optimized CPU library, with multi-thread help and a relaxed SIMD construct.
• GPU makes use of ML Drift, Google’s on-device GPU resolution, working via WebGPU.
• NPU makes use of the WebNN API, at the moment experimental in Chrome and Edge.

Two associated guidelines govern dispatch. First, LiteRT.js doesn’t help partial delegation. A graph can’t cut up throughout CPU and GPU.

Second, delegation is all-or-nothing per mannequin. If a mannequin can’t be absolutely delegated to the chosen accelerator, LiteRT falls again to wasm execution. The CPU path has the widest operator protection.

Efficiency

Given these backends, Google group stories two distinct outcomes.

Towards different internet runtimes, LiteRT.js is as much as 3x sooner throughout CPU and GPU inference. That determine covers classical pc imaginative and prescient and audio processing fashions.

Towards its personal CPU execution, GPU or NPU delivers a 5–60x speedup. That applies to demanding real-time work like object monitoring and audio transcription.

Each benchmarks ran in a managed browser surroundings on a 2024 MacBook Professional with M4 Apple Silicon. Google notes outcomes fluctuate with native GPU, thermal throttling, and driver optimization. A “10x” determine circulating alongside the launch doesn’t seem within the announcement.

Getting a PyTorch Mannequin In

LiteRT Torch converts PyTorch fashions to .tflite in a single step.

Nonetheless, the stipulations are strict. Your mannequin should be exportable with torch.export.export, that means TorchDynamo-exportable. It can’t comprise Python conditional branches that depend upon runtime tensor values. It additionally can’t have dynamic enter or output dimensions, together with the batch dimension.

For dimension, AI Edge Quantizer configures quantization schemes throughout totally different mannequin layers. Pretrained .tflite fashions are additionally accessible on Kaggle and the LiteRT Hugging Face Neighborhood.

The Minimal Pipeline

As soon as transformed, the runtime code is brief. That is the WebGPU path, verified towards @litertjs/core v2.5.2:

import {loadLiteRt, loadAndCompile, Tensor} from '@litertjs/core';

// Wasm information dwell in node_modules/@litertjs/core/wasm/ or on a CDN.
await loadLiteRt('path/to/wasm/listing/');

const mannequin = await loadAndCompile('path/to/mannequin.tflite',  'webnn'
);

const enter = new Tensor(new Float32Array(1 * 3 * 224 * 224), [1, 3, 224, 224]);
const outcomes = await mannequin.run(enter);

// Accelerator outcomes dwell off-heap. Transfer to CPU, then convert.
const cpuTensor = await outcomes[0].moveTo('wasm');
const output = cpuTensor.toTypedArray();

// LiteRT.js makes use of handbook reminiscence administration. Delete each tensor.
enter.delete();
for (const t of outcomes) t.delete();
cpuTensor.delete();

That final block deserves consideration. LiteRT.js does not garbage-collect tensors. Each Tensor should be deleted explicitly, or the app leaks machine reminiscence. The snippet in Google’s announcement publish omits this step.

WebNN wants one additional flag. LiteRT.js requires JSPI, which bridges synchronous kernel scheduling with asynchronous machine polling:

await loadLiteRt('path/to/wasm/', {jspi: true});

const mannequin = await loadAndCompile('mannequin.tflite', {
  accelerator: 'webnn',
  webNNOptions: {devicePreference: 'npu'}, // 'cpu' | 'gpu' | 'npu'
});

Earlier than writing pre-processing, take a look at with faux inputs. Run npm i @litertjs/model-tester, then npx model-tester. It runs your mannequin on WebNN, WebGPU, and CPU utilizing random inputs. Use mannequin.getInputDetails() to learn enter names and shapes.

Use Circumstances With Examples

These APIs again 4 demos Google shipped at launch:

LiteRT.js vs TensorFlow.js

These demos increase an apparent query for current internet ML groups.

Importantly, the 2 should not mutually unique. Google positions LiteRT.js as a substitute for TF.js Graph Fashions particularly, not the entire library.

TensorFlow.js stays the really helpful device for pre- and post-processing. The @litertjs/tfjs-interop bundle passes tensors between them through runWithTfjsTensors. Keep away from tensor.dataSync, which carries a major penalty on the WebGPU backend.

Interactive Explainer

The embed under animates the six pipeline phases throughout every backend.