Server-sent Events
Large language models can take a long time to generate data; even longer if they are used as part of an agentic loop. Server-sent events (SSE) are the standard way to stream data between your LLM provider, your server, and your browser client: While SSEs are easy-to-use and simple to understand, production applications often run into issues. Specifically:1. SSEs break over flaky connections
If your client’s long-running SSE connection drops due to network issues, they can’t reconnect to the existing stream. If the client reconnects, they typically won’t be able to load any of that in-flight data. The client typically has to wait for you to store the full LLM output in your database, then refetch it to see the result.2. SSEs can’t be sent from async jobs
Often, you’ll want to trigger the LLM inference from an async job instead of directly on the server. Async jobs offer reliability by processing their requests in the background after the server has disconnected from the client. Unfortunately, there’s no direct way to stream to the client to update their UI from an async job. You can write to a database instead, but you would need to set up a stream to the client from the database. Additionally, LLMs stream each chunk faster than they can be written to most databases.3. SSEs are limited to one stream per HTTP request
If you’re generating multiple responses from a LLM, you’ll need to figure out a way to mux the responses into a single SSE stream, adding complexity. Some SDKs will do this for you, but not all support this. Additionally, SSEs are only accessible in the tab that created the network request. It’s impossible to stream a SSE to two tabs open to the same page.WebSockets
WebSockets create a long-running, free-form events channel between a client and server, supporting bi-directional communication. A popular abstraction for websockets, socket.io, offers reliable, in-order, buffered delivery of events. However, WebSockets don’t fully solve the problem.1. WebSockets can’t replay past events
If a page reloads during a WebSocket stream and the events are not persisted on the client, those events are lost. Socket.io will indeed buffer events when the connection is flaky, but it does not store past events—only undelivered ones. To solve this problem, there needs to be a persistent storage layer that temporarily stores events, so that they can be replayed when a connection is reestablished. Streamstraight solves this by persisting events to Redis.2. WebSockets also can’t be sent from async jobs
Calling LLMs from an async jobs allows for greater reliability, but because a WebSocket is a connection between a client and a server, WebSockets cannot be sent directly from async jobs. You can write to a database instead, but LLMs stream each chunk faster than they can be written to most databases. There needs to be a middleman that the async job can stream, which then can be forwarded to the client. Streamstraight is designed to be this middleman.3. WebSockets require custom implementation
WebSockets are a barebones protocol that supports any kind of event-driven communication. Using them specifically for streaming LLM output requires abstraction to handle chunks, acknowledgments, and reliability. Even if you were to use socket.io, there’s a good amount of additional custom implementation to make it work:- Adding a persistent storage layer (Redis, Cloudflare Durable Objects, etc.) for replaying streams
- Avoiding contention between connections to the storage layer
- Properly incrementing cursors and tailing chunks
- Properly authenticating your client with your WebSockets server
- Converting between different stream formats and libraries: async events,
ReadableStream,AsyncIterable, AI SDK Transports, etc.