gRPC Services in Kotlin: Schema-First Design, Streaming, and Kubernetes Load Balancing

gRPC is a high-performance remote procedure call framework built on HTTP/2 and Protocol Buffers. In a microservices architecture, it’s the natural choice for synchronous service-to-service communication when you need strong typing, efficient serialization, bidirectional streaming, or per-language code generation from a single schema.

This guide covers the order service gRPC implementation:

• the schema-first design workflow
• Kotlin coroutine-based server implementation
• interceptors
• streaming patterns
• and the tricky problem of load balancing gRPC in Kubernetes.

gRPC vs. REST for Internal Services

REST is the right choice for public-facing APIs. JSON is human-readable, HTTP/1.1 is universally understood, and tooling is ubiquitous. But for internal service-to-service communication, these properties matter less, and gRPC’s advantages become significant:

Protocol Buffers

Protocol Buffers serialize to compact binary, typically 5–10x smaller than equivalent JSON. For high-volume internal calls, this reduces bandwidth and deserialization CPU cost.

Strong typing

A shared schema means the compiler catches breaking changes before deployment, not in production. A REST API returning a renamed field fails silently at runtime.

Code generation

Code generation from .proto files produces client and server stubs in every major language. Adding a new service in Go that calls the Kotlin order service means running protoc — not writing an HTTP client from scratch.

HTTP/2 multiplexing

HTTP/2 multiplexing allows multiple concurrent RPCs over a single TCP connection. For internal services making many parallel calls to the same backend, this significantly reduces connection overhead.

Native streaming

Streaming is a first-class concept in gRPC, not an afterthought. Server-sent events and WebSocket hacks are unnecessary.

Schema-First Design

Everything starts with the .proto file, which defines the contract between client and server. The order service proto defines the service with both unary and server-streaming RPCs:

syntax = "proto3";
package orders.v1;

service OrderService {
  // Unary: create one order, get one response
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);

  // Server-streaming: request once, receive a stream of orders
  rpc ListOrders(ListOrdersRequest) returns (stream Order);
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated OrderItem items = 2;
  string idempotency_key = 3;
}

message Order {
  string order_id = 1;
  string customer_id = 2;
  repeated OrderItem items = 3;
  OrderStatus status = 4;
  int64 created_at_unix_ms = 5;
  int64 total_cents = 6;
}

enum OrderStatus {
  ORDER_STATUS_UNKNOWN = 0;
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_CONFIRMED = 2;
  ORDER_STATUS_CANCELLED = 3;
}

The field numbers (= 1, = 2) are what actually get serialized in the binary format — not the field names. This means you can rename fields without breaking backward compatibility, as long as you keep the same numbers. But changing a field number breaks all existing serialized data. The rule: field numbers are forever.

Adding new fields with new numbers is safe (old clients ignore unknown fields). Removing fields requires caution — mark them reserved to prevent the number from being reused later.

Kotlin gRPC Server: Coroutines over Callbacks

gRPC-Kotlin generates coroutine-based stubs instead of the callback or ListenableFuture style used by the Java SDK. The difference is significant: suspend fun for unary calls and Flow<T> for streaming calls read like ordinary sequential code.

class OrderServiceImpl(
    private val store: OrderStore = OrderStore(),
) : OrderServiceGrpcKt.OrderServiceCoroutineImplBase() {

    override suspend fun createOrder(request: CreateOrderRequest): CreateOrderResponse {
        require(request.customerId.isNotBlank()) { "customerId must not be blank" }
        require(request.itemsList.isNotEmpty()) { "Order must have at least one item" }

        val totalCents = request.itemsList.sumOf { it.unitPriceCents * it.quantity }
        val newOrder = order {
            orderId = UUID.randomUUID().toString()
            customerId = request.customerId
            items.addAll(request.itemsList)
            status = OrderStatus.ORDER_STATUS_PENDING
            createdAtUnixMs = System.currentTimeMillis()
            this.totalCents = totalCents
        }
        store.save(newOrder)
        return createOrderResponse { order = newOrder }
    }

suspend fun createOrder is a unary RPC — the client sends one request, the server returns one response. The order { ... } block uses Kotlin DSL builders generated by protoc-gen-kotlin, which are considerably more ergonomic than the Java builder pattern.

Idempotency

The comment in the code calls out an important production concern:

// Idempotency: in production, look up request.idempotencyKey in a cache/DB before
// creating. If found, return the existing order. This ensures safe retries without
// creating duplicate orders.

gRPC clients (and the Istio mesh) will retry failed calls. Without idempotency handling, a network blip that drops the response — not the request — can create duplicate orders. The idempotency_key field in CreateOrderRequest exists specifically to support this: store the key alongside the created order, and on retry, return the existing order instead of creating another.

Server-Streaming RPC

override fun listOrders(request: ListOrdersRequest): Flow<Order> {
    val candidates = if (request.customerId.isNotBlank()) {
        store.findByCustomer(request.customerId)
    } else {
        store.findAll()
    }

    val maxResults = if (request.maxResults > 0) request.maxResults else 100

    return candidates
        .filter { order ->
            statusFilter == OrderStatus.ORDER_STATUS_UNKNOWN || order.status == statusFilter
        }
        .take(maxResults)
        .asFlow()
}

listOrders returns Flow<Order>. gRPC-Kotlin sends each emitted value as a separate streaming message. The client receives orders incrementally — it can start processing the first order before the last one arrives. For large result sets, this eliminates the “load everything into memory, serialize, send” pattern.

The filtering and take(maxResults) in the Flow pipeline execute lazily — the client fully processes only the orders it actually consumes. In a production implementation backed by a database, this would push the filtering into the query rather than loading all records in memory.

The Four Streaming Patterns

gRPC supports four communication patterns:

This implementation uses unary (CreateOrder) and server-streaming (ListOrders). Bidirectional streaming is the most powerful but also the most complex — it requires careful handling of backpressure, half-close semantics, and error propagation.

Interceptors: Cross-Cutting Concerns Without Cluttering Business Logic

class LoggingInterceptor : ServerInterceptor {
    override fun <ReqT : Any, RespT : Any> interceptCall(
        call: ServerCall<ReqT, RespT>,
        headers: Metadata,
        next: ServerCallHandler<ReqT, RespT>,
    ): ServerCall.Listener<ReqT> {
        val method = call.methodDescriptor.fullMethodName
        val start = System.currentTimeMillis()
        log.info("gRPC call started: {}", method)
        return object : ForwardingServerCallListener.SimpleForwardingServerCallListener<ReqT>(
            next.startCall(
                object : ForwardingServerCall.SimpleForwardingServerCall<ReqT, RespT>(call) {
                    override fun close(status: Status, trailers: Metadata) {
                        val elapsed = System.currentTimeMillis() - start
                        log.info("gRPC call finished: {} status={} elapsed={}ms",
                            method, status.code, elapsed)
                        super.close(status, trailers)
                    }
                },
                headers,
            )
        ) {}
    }
}

The interceptor pattern in gRPC works the same way as HTTP middleware: wrap the call, do something before and after, forward to the next handler. The production extension points are trace context propagation (W3C traceparent or B3 headers from Metadata), request duration as a Prometheus histogram, and JWT validation if not delegated to the service mesh.

The server wires everything together:

fun buildServer(port: Int): Server {
    val serviceImpl = OrderServiceImpl()
    val interceptedService = ServerInterceptors.intercept(serviceImpl, LoggingInterceptor())

    return ServerBuilder
        .forPort(port)
        .addService(interceptedService)
        .build()
}

Multiple interceptors apply in reverse registration order (last registered = first executed), matching the middleware convention in most frameworks.

Load Balancing gRPC in Kubernetes: The HTTP/2 Problem

HTTP/1.1 opens a new TCP connection per request (or reuses connections briefly). Kubernetes Services work well with this model because kube-proxy can route each new connection to a different pod.

gRPC uses HTTP/2, which multiplexes many RPCs over a single long-lived TCP connection. A client that connects to orders-service via a ClusterIP opens one connection to one pod and sends all RPCs over it, even if there are 10 replicas. kube-proxy only sees the connection at establishment time — it doesn’t balance individual RPCs.

Solutions:

Client-side load balancing: The gRPC client resolves DNS, gets all pod IPs (using a headless Service), and balances across them itself. gRPC has a built-in round_robin policy. This works but requires the client to handle DNS refresh and pod lifecycle.

Proxy-based load balancing (preferred): A proxy that understands HTTP/2 (Envoy, Istio, Linkerd) can load balance individual gRPC calls — not just connections. The service mesh pattern uses Istio’s Envoy sidecar to handle this transparently without changing application code.

gRPC keepalive: Even with proxy balancing, configure gRPC keepalive pings to detect dead connections without waiting for a request to fail. Set KEEPALIVE_TIME_MS on both client and server.

For this Kotlin service running in Kubernetes with Istio, the service mesh handles gRPC load balancing. The application code makes no special accommodations — it exposes a standard gRPC server on port 9090, and the mesh handles the rest.

Key Takeaways

• gRPC is the right choice for internal services when you need strong typing, efficient serialization, or streaming — REST is better for public-facing APIs
• Field numbers in .proto files are permanent — you can rename fields but never reuse a number
• idempotency_key fields are essential for safe retry semantics — store the key and return the existing result on duplicate requests
• gRPC-Kotlin generates suspend fun for unary calls and Flow<T> for streaming — no callbacks or futures needed
• Server-streaming RPCs (returning Flow<T>) let clients receive large result sets incrementally without buffering everything in memory
• Interceptors handle cross-cutting concerns (logging, tracing, auth) without cluttering service implementations
• HTTP/2 multiplexing means kube-proxy cannot load balance gRPC at the RPC level — use a service mesh (Istio/Envoy) or client-side load balancing with a headless Service
• grpcurl -plaintext localhost:50051 list is your first debugging tool for a running gRPC server