Forget JSON. These 4 Data Formats Made Our API 5x Faster

Your JSON Looks Fine. So Why Is Everything Slow?

I was staring at response times on a service in production when something didn't add up. DB queries were fast. The response was not. The logic was trivial. When I finally cornered the culprit, it was serialization — turning objects into JSON strings on one end, parsing them back into objects on the other.

Nobody thinks about that step. JSON is comfortable. It renders in the browser console, a teammate can open it in Notepad and understand it, and debugging means just looking at it. That comfort is so strong that most people stop right there.

Comfort has a price tag.

Every time your server serializes a response, it takes structured in-memory data and flattens it into a string, character by character. The client receives that string, parses it, rebuilds the structure. Once or twice, you'd never notice. In production this happens thousands of times per second. And people still conclude "it runs fine."

It does not run fine.

Here are real numbers from my own service. A payload of 10,000 user records as JSON came out to 2.3MB and took roughly 180ms to serialize. Same data in a binary format? 620KB, about 34ms.

Five times faster on identical data. I didn't change a single character of the content. I only changed the container. Here's what I changed and how.

1. Protobuf — Send Field Numbers, Not Field Names

Google ran this internally for years before open-sourcing it. The idea is simpler than the reputation suggests: define your schema once in a .proto file, and the library handles encoding and decoding into a fixed binary format.

// user.proto
syntax = "proto3";

message User {
  int32 id = 1;
  string name = 2;
  string email = 3;
}

Encoding and decoding in Node.js:

const protobuf = require('protobufjs');

async function run() {
  const root = await protobuf.load('user.proto');
  const User = root.lookupType('User');
  const msg = User.create({ id: 1, name: 'Arjun', email: 'a@dev.io' });
  const buf = User.encode(msg).finish(); // 바이너리 버퍼 생성
  const decoded = User.decode(buf);      // 객체로 변환
}

The key detail: what goes over the wire is field numbers (1, 2, 3). Not field names. In JSON, the string "name" eats 4 bytes on every single response. In Protobuf, that same information is one byte of field tag.

Sounds like nothing. Now imagine a million requests a day. Three bytes per field, ten fields, thirty bytes per request. A million requests and you're evaporating tens of megabytes into the network every day. That's not micro-optimization, that's an infrastructure line item — it touches transfer fees, bandwidth, and CPU cycles directly.

Where it fits: internal service-to-service traffic, gRPC APIs, high-throughput data pipelines. If both ends can share a schema, this is almost always the answer.

2. MessagePack — JSON That Went to the Gym

Think of MessagePack as JSON after it lost the weight.

It's still key-value. It supports basically the same data types. The one difference is that everything encodes as binary instead of text. No schema required. You can serialize your existing JSON objects with close to zero migration work.

I'll be honest: I tried this one first because I didn't have the patience to write Protobuf schemas. Converting our WebSocket payloads from JSON to MessagePack took one afternoon. I didn't touch a single line of business logic, and payload size dropped 38%.

const msgpack = require('@msgpack/msgpack');

const data = { id: 1, name: 'Arjun', score: 98.5 };
const encoded = msgpack.encode(data); // Uint8Array, 약 20바이트
const decoded = msgpack.decode(encoded); // 다시 일반 객체로

Run that same object through JSON.stringify(data) and you get {"id":1,"name":"Arjun","score":98.5} — 37 bytes in ASCII. MessagePack does it in under 20.

How? MessagePack represents common values like small integers, booleans, and null as single bytes instead of spelling them out. true costs one byte, not four. It adds up.

There's a cost, though. You lose readability. Before you convert anything, verify that your logging and debugging pipelines can handle binary data. Skip that step and you'll regret it during your first incident. More on this later.

Where it fits: WebSocket payloads, Redis caching — anywhere you control both ends of the wire. For anything that never leaves your own systems, this has the lowest barrier to entry.

Where Formats Belong — It's a Boundary Problem

Here's the mental model that finally sorted this out for me. The format should depend on whether a human or a machine is reading the data. That's the whole rule.

클라이언트 앱
     |
     | (REST/HTTP - JSON. 사람이 직접 디버깅하는 구간)
     v
API 게이트웨이
     |
     | (MessagePack 또는 Protobuf - 시스템 전용)
     v
서비스 A ----> 서비스 B ----> 서비스 C
     |                          |
     | (Avro - 이벤트 스트리밍)   |
     v                          v
Kafka / Redpanda            캐시 (MessagePack)

The boundaries matter. Public APIs live or die on readability. Internal calls live or die on speed. Most teams lay JSON across every hop and then wonder, on repeat, why things are slow.

Separate cause from effect and it gets obvious. Slow is the effect. The cause is using a human-readable format on internal hops no human will ever read. And the answer lives in one line: the Content-Type header. That's exactly where you decide which format goes where.

I think of it as the system's skin and its organs. Skin touches the outside world, so people poke at it and look at it. JSON is right there. Organs run fast on the inside and nobody looks at them directly. Using a text format there is like labeling your internal organs.

3. Apache Avro — Schemas Change, Nothing Breaks

If you've ever built a pipeline on Kafka, you end up here eventually. Nobody tells you up front, but this is the format Kafka is quietly expecting.

It's schema-based like Protobuf, with one decisive difference: the schema is stored alongside the data. That makes it especially good at event streaming, where schemas drift over time.

Think about it. You added three fields last week. A consumer reading events written six months ago still has to work. If a schema change makes your old data unreadable, that's not a data lake, that's a data graveyard. Avro handles this schema evolution cleanly.

const avro = require('avsc');

const UserEvent = avro.Type.forSchema({
  type: 'record',
  name: 'UserEvent',
  fields: [
    { name: 'id', type: 'int' },
    { name: 'event', type: 'string' },
    { name: 'ts', type: 'long' }
  ]
});

const buf = UserEvent.toBuffer({ id: 42, event: 'login', ts: Date.now() });
const obj = UserEvent.fromBuffer(buf);

The numbers: after moving our internal event bus to Avro, Kafka consumer lag dropped from 40 seconds at peak load to under 4 seconds. Same hardware. Same topic partitions. The only change was the wire format.

I didn't believe that number when I first saw it. A 10x improvement is usually a measurement error. But it held across several more days of watching. The bottleneck keeping consumers from catching up was serialization and deserialization cost, plain and simple.

Confluent, AWS Glue — wire it into a schema registry and a team running pipelines at scale can pick it up immediately.

Where it fits: Kafka event streaming, data lakes, anywhere schema change over time is a real design concern.

4. FlatBuffers — Read the Box Without Opening It

The most underrated and most misunderstood format on this list.

Also from Google, but the approach is completely different. Other formats encode objects into bytes and decode them on the far side. FlatBuffers deletes the decode step entirely. It lays out memory from the start so your code can read the byte buffer directly.

The analogy: with other formats you get a package, tear off the wrapping, and pull out the contents. FlatBuffers never opens the box. It reads the box itself. Deserialization as a step doesn't exist.

// .fbs 스키마에서 JS 코드를 생성한 후:
const flatbuffers = require('flatbuffers');
const { Monster } = require('./monster_generated');

const builder = new flatbuffers.Builder(128);
const name = builder.createString('Orc');
Monster.startMonster(builder);
Monster.addHp(builder, 300);
Monster.addName(builder, name);
const orc = Monster.endMonster(builder);
builder.finish(orc);
const buf = builder.asUint8Array();

const monster = Monster.getRootAsMonster(new flatbuffers.ByteBuffer(buf));
console.log(monster.name()); // 'Orc' - 버퍼에서 직접 읽기, 제로 카피
console.log(monster.hp());   // 300

No copying. No memory allocation on read. For latency-obsessed systems — live feeds, financial trade data, game backends — that's a different class of performance. When you get to the benchmark below, the FlatBuffers deserialization number will look like a typo. It isn't.

The barrier is real too. You write a schema (.fbs) and run a code generation step. The code that builds your data is more verbose than the alternatives. So it's not "it's fast, use it everywhere" — it's a targeted pick for places where the last few milliseconds are worth money.

Where it fits: real-time systems, financial data feeds, game servers, embedded systems.

Benchmarks — Actual Numbers

Measured on Node.js 20 with a 5,000-record payload, 8 fields per record.

형식	크기	직렬화	역직렬화
JSON	1.8 MB	142 ms	98 ms
MessagePack	1.1 MB	61 ms	44 ms
Protobuf	680 KB	38 ms	29 ms
Avro	590 KB	35 ms	31 ms
FlatBuffers	720 KB	28 ms	~2 ms*

*FlatBuffers deserialization is near zero because there is no deserialization. Your code reads straight from the raw buffer in memory.

Read the table structurally and it gets interesting. Avro wins on size (590KB). FlatBuffers wins on serialization (28ms) and dominates on deserialization (~2ms). There's no format that takes first place across the board. The answer depends on what you're trying to shrink.

Cutting network cost by cutting size? Avro or Protobuf. Driving read latency toward zero? FlatBuffers. Minimizing code changes? MessagePack. That's the real selection criterion — not picking the benchmark winner, but looking at where your bottleneck actually is first.

So Should You Drop JSON Entirely?

No. That's the wrong conclusion. I spent a whole weekend trying to talk myself into "everything should be binary" and failed.

JSON is still correct for public APIs, config files, and anywhere a human has to read the data. Debugging convenience is an enormous asset on its own, and the tooling ecosystem support is overwhelming. Ignore that and push everything to binary and you're grinding up development velocity and operational stability to pay for it.

Now for the thing I deferred earlier. Your first production incident after the switch, you open the logs and get an unreadable blob of binary. Live through that once and you'll know exactly what I mean. Squinting at an unknown byte array at 3 a.m. is not compensated by a 38% smaller payload.

So the decision rule, kept simple:

1. Humans read and debug it? JSON. Public APIs, config files, webhook payloads.
2. Internal service traffic where both sides can share a schema? Protobuf. Even more natural alongside gRPC.
3. Want a fast win with no code changes and you own both ends? MessagePack. Redis, WebSockets.
4. Event streaming with schemas that drift over time? Avro. It's close to the default in the Kafka ecosystem.
5. Real-time system where the last 1ms is money? FlatBuffers. Financial feeds, game servers.

What Actually Matters Is Intent

Compress all of this into one line: the real change isn't switching formats, it's learning to choose a format on purpose.

JSON isn't slow. The habit of laying JSON across every boundary without thinking is slow. JSON where humans touch the system, binary where machines talk to each other in bulk. Being conscious of that single distinction changes the character of a system.

In consulting work I saw a lot of systems that "just don't perform," and a surprising number of them had blurry data-format boundaries. Sometimes checking Content-Type gets you further, faster, than editing code. The judgment about what goes where comes before swapping tools.