I nearly shipped a nasty AI integration as soon as. The function labored technically: the mannequin answered questions appropriately, the UI seemed clear, and everybody nodded alongside in our inside demo. Then we put actual customers on it, and inside 48 hours a product supervisor Slack’d me asking why 60% of classes have been ending with no response obtained.
The issue wasn’t the API or the server. It was eight seconds of silence.
Customers despatched a message, noticed nothing occur, assumed the request had failed, and closed the tab. The response would arrive shortly after, to no one. We had constructed a practical AI function that felt utterly damaged, and the repair wasn’t an architectural overhaul or a efficiency optimization. It was streaming: begin sending tokens the second the mannequin produces them, as a substitute of batching the total response and delivering it without delay.
I’ve seen this error in codebases throughout a number of firms now. The non-streaming model is easier to put in writing, checks go, and no one notices in improvement as a result of builders watch for issues, whereas actual customers often don’t.
This tutorial builds the entire streaming implementation: a Node.js backend utilizing Server-Despatched Occasions to push tokens to the browser as they arrive, with session administration that provides the mannequin reminiscence throughout turns, and correct abort dealing with so you aren’t paying for tokens no one reads.
The entire code is at github.com/ziaongit/nodejs-openai-streaming.
No OpenAI key? I examined this whole article utilizing Groq as a substitute. It’s free, it speaks the identical OpenAI SDK format, and getting a key takes about two minutes. The repo has a
USE_GROQ=trueflag already wired in. Copy.env.instanceto.env, paste your Groq key, and you might be working. I usedllama-3.3-70b-versatile. The output is nice sufficient that you wouldn’t discover the distinction in a dev session.
Stipulations
- Node.js v18.11 or later
- An OpenAI API key
- Stable familiarity with Specific and async/await
The Structure Earlier than Any Code
Earlier than we write any code, it helps to see what is definitely taking place throughout the three layers. Debugging streaming points with out this image will get painful quick.
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ 1. POST /session ─────────────────────► receives sessionId │
│ 2. POST /chat { message, sessionId } │
│ 3. response.physique.getReader() ──► ReadableStream │
│ 4. decode chunks ──► parse SSE occasions ──► append tokens to UI │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTP (textual content/event-stream)
▼
┌──────────────────────────────────────────────────────────────────┐
│ Specific Server │
│ │
│ classes Map ── { sessionId: { messages: […] } } │
│ │
│ POST /chat │
│ ├─ load session historical past │
│ ├─ push person message │
│ ├─ set SSE headers + flushHeaders() │
│ ├─ openai.create({ stream: true, sign: controller.sign }) │
│ ├─ for await chunk → res.write(`knowledge: ${token}nn`) │
│ ├─ accumulate fullResponse → push to session │
│ └─ ship { accomplished: true } → res.finish() │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTPS chunked switch encoding
▼
┌──────────────────────────────────────────────────────────────────┐
│ OpenAI API │
│ gpt-4o-mini · stream: true │
└──────────────────────────────────────────────────────────────────┘
The session retailer lives on the server. Each time the person sends a message, that full dialog historical past goes to OpenAI contained in the messages array, which is how the mannequin is aware of what was mentioned earlier. The API itself remembers nothing between calls, so that you carry the state.
The mechanism is easier than it sounds: open an everyday HTTP response and by no means shut it. The server retains writing chunks into the socket as tokens arrive, and when the mannequin finishes you name res.finish() and the connection drops.
Undertaking Setup
mkdir nodejs-openai-streaming
cd nodejs-openai-streaming
npm init -y
npm set up specific openai cors dotenv
mkdir public
Add `"sort": "module"` to `bundle.json`:
{
"identify": "nodejs-openai-streaming",
"model": "1.0.0",
"sort": "module",
"scripts": {
"begin": "node server.js",
"dev": "node --watch server.js"
},
"dependencies": {
"cors": "^2.8.6",
"dotenv": "^16.6.1",
"specific": "^4.22.2",
"openai": "^4.104.0"
}
}
Create .env:
Take a look at our hands-on, sensible information to studying Git, with best-practices, industry-accepted requirements, and included cheat sheet. Cease Googling Git instructions and truly study it!
OPENAI_API_KEY=your_api_key_here
PORT=3000
Undertaking construction:
nodejs-openai-streaming/
├── server.js
├── public/
│ └── index.html
├── .env
├── .env.instance
├── .gitignore
└── bundle.json
Why SSE and Not WebSockets
Individuals ask about WebSockets each time, often as a result of they sound extra succesful and due to this fact extra appropriate.
For AI chat, they often usually are not. Responses journey a method, server to consumer, and SSE is constructed for precisely that. It runs over plain HTTP with no protocol improve and no particular reverse proxy configuration, and it really works via HTTP/2 with out touching a config file. I’ve shipped SSE to manufacturing a number of occasions and by no means as soon as nervous about browser assist.
WebSockets make sense when the consumer must push knowledge again on the similar frequency the server pushes it: gaming, shared whiteboards, stay bidding. In an AI chat, the person varieties one message and sits there studying, so SSE suits the site visitors sample higher.
One real limitation of SSE value realizing: the browser’s built-in EventSource API solely helps GET requests. Since it is advisable to put up a message physique, you utilize fetch with a ReadableStream reader as a substitute. The SSE wire protocol is equivalent; you might be simply studying the stream manually fairly than via the browser’s occasion wrapper.
Session Administration
The primary chatbot I constructed on high of this API had a bug I couldn’t determine for 2 days. Customers would ask a follow-up query and the mannequin would reply prefer it was listening to from them for the primary time, with no context or reminiscence of the sooner turns.
The API doesn’t retailer something between calls, so every request goes in chilly. The messages array is the way you repair that: pack the total dialog historical past into each request, person turns and assistant turns each, and the mannequin responds as if it remembers the entire thread. Depart it out and also you get that very same damaged expertise I shipped by chance.
const classes = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
perform getOrCreateSession(sessionId) {
if (!classes.has(sessionId)) {
classes.set(sessionId, {
messages: [],
lastActive: Date.now(),
});
}
const session = classes.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of classes) {
if (now - session.lastActive > SESSION_TTL_MS) {
classes.delete(id);
}
}
}, 10 * 60 * 1000);
The Map is ok at improvement scale and in single-process deployments. The lastActive timestamp will get bumped on each request. The cleanup interval runs each ten minutes and removes classes which have gone quiet for half an hour.
The exhausting restrict of this method is {that a} Node.js course of restart wipes it completely. If you happen to run a number of cases behind a load balancer, every occasion has its personal Map, and a person whose requests hit completely different cases will lose their dialog mid-sentence. The repair is Redis: one consumer, similar knowledge construction, TTL dealt with natively. I’m not going to construct that right here as a result of it might double the size of this text with out including something about streaming, however don’t ship this to manufacturing with out it.
The Streaming Endpoint
4 issues occur in sequence earlier than tokens begin flowing: enter validation, SSE setup, abort dealing with, and the OpenAI name with its token loop. Every step has at the very least one non-obvious determination.
SSE Setup
res.setHeader('Content material-Sort', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
flushHeaders() is the road I see lacking in additional streaming PRs than I can depend. Skip it and Specific holds onto the response headers till the primary res.write() name. The browser opens the request, will get nothing again, sits there, and after a couple of seconds the entire thing seems like a hung request. You’ll spend an hour checking your OpenAI name, your async logic, and your middleware, and the repair is one line on the high. Name flushHeaders() proper after setting the headers; nothing else modifications.
AbortController
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
One necessary element: connect the shut listener to res, not req. The req object fires shut when the request physique is absolutely consumed, which occurs instantly after the POST physique arrives and would abort the stream earlier than it begins. res.on('shut') fires when the precise response connection drops, which is what you need. The !res.writableEnded guard prevents calling abort() after a clear end.
When the connection drops, controller.abort() propagates a cancellation sign to the OpenAI SDK and the for await loop exits on the subsequent iteration. With out this, you retain pulling tokens from the API and writing them right into a closed socket, which suggests you might be spending cash and reaching nothing.
The OpenAI Streaming Name
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{
role: 'system',
content: 'You are a helpful assistant. Be concise and clear.',
},
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
Set max_tokens. This isn’t elective. I left it uncapped throughout a testing session and watched my utilization dashboard spike in actual time. Left alone, the mannequin will write a dissertation in response to a three-word query. I exploit 800 for chat, which works fantastic for many issues, and you’ll tune it when you see what your customers truly ship.
The sign parameter threads the AbortController via to the SDK. If the consumer disconnects, the SDK cancels the underlying HTTP request to OpenAI, not simply the native loop.
Token Loop and Historical past
let fullResponse = '';
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`knowledge: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
}
Every chunk accommodates one token in delta.content material. You concatenate them into fullResponse as they arrive, relay each to the consumer, and save the entire string to the session as soon as the loop ends.
The double newline in res.write() is obligatory; it’s how SSE indicators the top of a single occasion. The browser buffers knowledge till it sees nn earlier than firing. Omit it and the consumer receives all tokens without delay when the connection closes, which is identical damaged habits as not streaming in any respect.
Don’t save fullResponse if the stream was aborted. A truncated assistant reply within the session historical past will corrupt the dialog context, and the mannequin’s subsequent flip can be constructed on incomplete data.
The Full server.js
import 'dotenv/config';
import specific from 'specific';
import cors from 'cors';
import OpenAI from 'openai';
import path from 'path';
import crypto from 'crypto';
import { fileURLToPath } from 'url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const app = specific();
const openai = new OpenAI({ apiKey: course of.env.OPENAI_API_KEY });
app.use(cors());
app.use(specific.json());
app.use(specific.static(path.be part of(__dirname, 'public')));
const classes = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
perform getOrCreateSession(sessionId) {
if (!classes.has(sessionId)) {
classes.set(sessionId, { messages: [], lastActive: Date.now() });
}
const session = classes.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of classes) {
if (now - session.lastActive > SESSION_TTL_MS) classes.delete(id);
}
}, 10 * 60 * 1000);
app.put up('/session', (req, res) => {
const sessionId = crypto.randomUUID();
getOrCreateSession(sessionId);
console.log(`[/session] created: ${sessionId}`);
res.json({ sessionId });
});
app.delete('/session/:sessionId', (req, res) => {
classes.delete(req.params.sessionId);
res.json({ okay: true });
});
app.put up('/chat', async (req, res) => {
const { message, sessionId } = req.physique;
if (!message || typeof message !== 'string') {
return res.standing(400).json({ error: 'message is required' });
}
if (!sessionId || typeof sessionId !== 'string') {
return res.standing(400).json({ error: 'sessionId is required' });
}
const session = getOrCreateSession(sessionId);
session.messages.push({ function: 'person', content material: message });
console.log(`[/chat] obtained:`, { message, sessionId });
res.setHeader('Content material-Sort', 'textual content/event-stream');
res.setHeader('Cache-Management', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
const controller = new AbortController();
res.on('shut', () => {
if (!res.writableEnded) controller.abort();
});
let fullResponse = '';
strive {
const stream = await openai.chat.completions.create(
{
mannequin: 'gpt-4o-mini',
messages: [
{ role: 'system', content: 'You are a helpful assistant. Be concise and clear.' },
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ sign: controller.sign }
);
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.selections[0]?.delta?.content material;
if (token) {
fullResponse += token;
res.write(`knowledge: ${JSON.stringify({ token })}nn`);
}
}
if (fullResponse) {
session.messages.push({ function: 'assistant', content material: fullResponse });
console.log(`[stream complete] ${fullResponse.size} chars`);
}
res.write(`knowledge: ${JSON.stringify({ accomplished: true })}nn`);
res.finish();
} catch (err) {
if (err.identify === 'AbortError' || err.identify === 'APIUserAbortError' || err.message === 'Request was aborted.') {
res.finish();
return;
}
console.error('Stream error:', err.message);
res.write(`knowledge: ${JSON.stringify({ error: err.message })}nn`);
res.finish();
}
});
const PORT = course of.env.PORT || 3000;
app.hear(PORT, () => console.log(`Server working at http://localhost:${PORT}`));
Constructing the Frontend
The consumer has two jobs: handle the session lifecycle and render the stream. The session lifecycle is easy (create on load, delete on “New Chat”), and the stream rendering is the place most implementations get tripped up.
html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta identify="viewport" content material="width=device-width, initial-scale=1.0">
<title>AI Streaming Chattitle>
<model>
* { box-sizing: border-box; margin: 0; padding: 0; }
physique {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: #f0f2f5;
top: 100vh;
show: flex;
flex-direction: column;
align-items: middle;
padding: 1.5rem 1rem;
}
.chat-wrapper { width: 100%; max-width: 700px; show: flex; flex-direction: column; top: 100%; }
h1 { font-size: 1.2rem; shade: #333; margin-bottom: 1rem; }
#messages {
flex: 1; overflow-y: auto; background: white; border: 1px strong #e0e0e0;
border-radius: 10px; padding: 1rem; show: flex; flex-direction: column; hole: 0.75rem;
}
.message {
max-width: 85%; padding: 0.65rem 0.9rem; border-radius: 10px;
line-height: 1.6; font-size: 0.93rem; white-space: pre-wrap; word-break: break-word;
}
.person { background: #0070f3; shade: white; align-self: flex-end; }
.assistant { background: #f1f1f1; shade: #111; align-self: flex-start; }
.input-row { show: flex; hole: 0.5rem; margin-top: 0.75rem; align-items: flex-end; }
textarea {
flex: 1; padding: 0.7rem; border: 1px strong #ddd; border-radius: 8px;
font-size: 0.93rem; font-family: inherit; resize: none; top: 52px; overflow-y: auto;
}
button { padding: 0.65rem 1.2rem; border: none; border-radius: 8px; font-size: 0.93rem; cursor: pointer; }
#ship { background: #0070f3; shade: white; }
#ship:disabled { background: #aaa; cursor: not-allowed; }
#clear { background: #f1f1f1; shade: #555; }
.standing { font-size: 0.78rem; shade: #aaa; margin-top: 0.3rem; min-height: 1rem; }
model>
head>
<physique>
<div class="chat-wrapper">
<h1>AI Chath1>
<div id="messages">div>
<p class="standing" id="standing">p>
<div class="input-row">
<textarea id="enter" placeholder="Ask one thing... (Ctrl+Enter to ship)">textarea>
<button id="ship">Shipbutton>
<button id="clear">New Chatbutton>
div>
div>
<script>
let sessionId = null;
async perform initSession() {
const res = await fetch('/session', { methodology: 'POST' });
const knowledge = await res.json();
sessionId = knowledge.sessionId;
}
perform appendMessage(function, textual content) {
const div = doc.createElement('div');
div.className = `message ${function}`;
div.textContent = textual content;
doc.getElementById('messages').appendChild(div);
div.scrollIntoView({ habits: 'easy' });
return div;
}
doc.getElementById('ship').addEventListener('click on', async () => {
const enter = doc.getElementById('enter');
const message = enter.worth.trim();
if (!message || !sessionId) return;
const sendBtn = doc.getElementById('ship');
const standing = doc.getElementById('standing');
enter.worth = '';
sendBtn.disabled = true;
standing.textContent = 'Producing...';
appendMessage('person', message);
const assistantBubble = appendMessage('assistant', '');
strive {
const response = await fetch('/chat', {
methodology: 'POST',
headers: { 'Content material-Sort': 'utility/json' },
physique: JSON.stringify({ message, sessionId }),
});
if (!response.okay) {
const err = await response.json();
assistantBubble.textContent = `Error: ${err.error}`;
return;
}
const reader = response.physique.getReader();
const decoder = new TextDecoder();
whereas (true) {
const { worth, accomplished } = await reader.learn();
if (accomplished) break;
const strains = decoder.decode(worth, { stream: true }).break up('n');
for (const line of strains) {
if (!line.startsWith('knowledge: ')) proceed;
strive {
const knowledge = JSON.parse(line.slice(6));
if (knowledge.accomplished) break;
if (knowledge.error) { assistantBubble.textContent += `n[Error: ${data.error}]`; break; }
if (knowledge.token) {
assistantBubble.textContent += knowledge.token;
assistantBubble.scrollIntoView({ habits: 'easy' });
}
} catch { }
}
}
standing.textContent = '';
} catch (err) {
assistantBubble.textContent = `Request failed: ${err.message}`;
} lastly {
sendBtn.disabled = false;
standing.textContent = '';
enter.focus();
}
});
doc.getElementById('clear').addEventListener('click on', async () => {
if (!sessionId) return;
await fetch(`/session/${sessionId}`, { methodology: 'DELETE' });
await initSession();
doc.getElementById('messages').innerHTML = '';
doc.getElementById('standing').textContent = '';
});
doc.getElementById('enter').addEventListener('keydown', (e) => {
if (e.key === 'Enter' && e.ctrlKey) doc.getElementById('ship').click on();
});
initSession();
script>
physique>
html>
The strive/catch contained in the inside loop handles an actual edge case: SSE occasions often span two community reads. When that occurs, JSON.parse throws on the unfinished fragment, and catching it so the subsequent learn can ship the remaining retains the stream alive. With out the catch, the entire stream crashes on what is basically regular community habits.
Working It
npm run dev
The terminal confirms the server began:
Server working at http://localhost:3000
Hit http://localhost:3000 in your browser and kind one thing. The terminal ought to print:
[/session] created: 2dfb2586-4601-4781-932d-cb3dcca5d5a5
[/chat] obtained: {
message: 'Are you aware about StackAbuse?',
sessionId: '2dfb2586-4601-4781-932d-cb3dcca5d5a5'
}
[stream complete] 304 chars
Within the browser, you will note the response construct out token by token, which suggests the streaming is working. The interface seems like this with a stay response:
Ship a follow-up that references the primary reply and the mannequin will bear in mind it. The session historical past is included in each request, so context carries throughout turns. Click on “New Chat” and the server deletes the session and creates a recent one. The mannequin begins over with no reminiscence of the earlier dialog.
To examine the uncooked stream, open DevTools, go to Community, discover the /chat request, and have a look at the EventStream tab. You will note every knowledge: occasion arrive individually, which is essentially the most helpful factor I do know of for debugging streaming habits that appears appropriate within the UI however shouldn’t be working the way in which you suppose.
Manufacturing Gaps to Deal with Earlier than Launch
Earlier than you ship this, know what the implementation doesn’t deal with. I’ve seen builders take tutorial code and deploy it unchanged.
Context overflow. Each request sends the total session historical past. gpt-4o-mini has a 128,000-token context window, which is massive sufficient that almost all chat classes won’t ever hit it. However a session working for hours will ultimately method the restrict. When it does, the API returns a context_length_exceeded error. The usual repair is a sliding window: preserve the system immediate, trim outdated messages from the entrance of the historical past, optionally run a summarization name to compress outdated context earlier than it falls off. This can be a actual engineering drawback and there’s no one-size reply.
Price limiting. There’s none right here, so one consumer can exhaust your month-to-month OpenAI quota. Add express-rate-limit earlier than this goes anyplace close to the web.
Authentication. The /chat endpoint accepts requests from anybody. In an actual deployment, you want session-based auth or JWT validation earlier than the route handler runs. With out it, anybody who is aware of your URL is spending your API credit.
Multi-process classes. Coated above. Redis or equal earlier than you scale horizontally.
Conclusion
I’ve shipped variations of this precise setup to manufacturing twice now, throughout completely different firms and merchandise, with the identical core sample: session retailer on the server, SSE for transport, and AbortController to cease losing cash on disconnected shoppers. The gaps I listed usually are not hypothetical warnings; I hit most of them myself earlier than determining the fixes.
The half that journeys most builders shouldn’t be the OpenAI API itself, however the plumbing round it: getting tokens to the browser the second they arrive, holding the connection alive with out burning credit on disconnected shoppers, and sustaining dialog context with out letting it develop unbounded. That’s the place the true work sits. Hopefully this text saved you the afternoon I spent studying most of it the exhausting means.
The complete code is at github.com/ziaongit/nodejs-openai-streaming. Clone it, break it, and make it your personal.
