Go load testing before your first backend hire

Ten thousand monthly active users is not a capacity requirement. It is a billing or analytics number. A Go service can support far more than that when requests are light and spread across the month, or fall over with a few hundred users when each session fans out into slow queries, uploads, and third-party calls.

The useful question is whether the generated backend meets a stated service target under your expected peak traffic, with enough margin for growth and an ordinary failure. You can answer that before hiring a backend engineer, but only if the load test looks like your product and records what the API, Go runtime, and PostgreSQL are doing at the same time. A green average latency graph proves almost nothing.

This is the test I would require before telling a founder that a generated Go backend is ready for 10,000 monthly active users. It produces a repeatable pass or fail result, exposes the first bottleneck, and separates a capacity problem from a correctness problem.

Monthly users must become peak requests

Convert the user forecast into requests per second before choosing any load level. Monthly active users hide the two variables that drive a backend: how many sessions arrive in the busiest window and how much work each session creates.

Start with observed data if a private beta exists. Count sessions in the busiest 15 minutes, requests per session, and the route mix. If there is no traffic yet, write assumptions down where everyone can challenge them. For example, suppose 10,000 active users create eight sessions per month, 15 API requests per session, and 20 percent of daily traffic lands in the busiest hour. That produces about 8 requests per second on an average busy day. Launches, notifications, payroll deadlines, or a shared time zone can make the real peak several times larger.

Do not turn that arithmetic into false precision. Use it to define three test levels:

• Expected peak: the busiest load you currently predict.
• Growth peak: twice the expected peak, unless the business has a better forecast.
• Stress level: increase traffic until a service target fails or a resource saturates.

The expected and growth tests answer whether the planned launch has margin. The stress test tells you what breaks first and how failure appears to users. That last answer matters because a service that rejects excess work quickly is easier to operate than one that consumes every database connection and stalls unrelated routes.

Use an open workload model for the capacity test. An open model starts requests at a fixed arrival rate even when prior requests slow down. A closed model with a fixed number of virtual users often hides collapse: slower responses cause those users to send fewer new requests, so offered load falls exactly when the service struggles. Grafana's k6 documentation makes this distinction through arrival-rate executors and reports dropped_iterations when the generator cannot start scheduled work. Treat dropped iterations as a test-generator failure, not a server success.

Run each steady level for at least 30 minutes after warm-up. Five-minute tests miss connection churn, garbage collection cycles, cache eviction, background jobs, and gradual memory growth. Add a separate two-hour soak at expected peak once the shorter test passes.

Write the conversion as a small worksheet and keep every unit visible. Monthly users multiplied by sessions per user and requests per session gives monthly requests. Divide only after assigning traffic to operating days and the busiest hour. Then add retry traffic, background jobs, webhooks, and polling that user analytics may not count. A frontend that polls every ten seconds can create more API work than the clicks that opened the page.

Model bursts separately from the steady peak. Login after a notification, an import finishing, or clients retrying after a short outage can concentrate work into a minute. Add a burst stage that reaches the expected burst rate quickly, holds it long enough to fill queues, and returns to normal. The service should recover without a growing backlog or a manual restart. Record recovery time as a result. A system can pass the steady test yet remain unsafe if a brief burst leaves its pool or workers stuck.

Do not multiply the final rate by an arbitrary safety factor and call the result realistic. Tie margin to a business uncertainty: forecast error, a planned campaign, one replica unavailable, or the time needed to add capacity. Test each assumption you intend to rely on. Keep the worksheet beside the result, because a passing run loses its meaning when nobody remembers which forecast and burst assumptions produced the target.

The traffic mix has to resemble a real session

A realistic load test preserves route frequency, payload size, authentication, data distribution, think time, and write contention. Hitting /health 100 times per second measures the health handler, not the application.

Build the mix from access logs when possible. Group routes by business action rather than raw URL, because /projects/123 and /projects/456 are one shape. A plausible early SaaS mix might allocate 45 percent to list and detail reads, 20 percent to search, 15 percent to creates or updates, 10 percent to login and token refresh, and 10 percent to exports or other heavy work. Your numbers should come from the product flow, not from that example.

Use many test accounts and records. Reusing one account can create an unrealistically hot cache, serialize updates on one row, or trigger rate limits that genuine traffic would spread out. Seed small, medium, and large tenants. Include missing records, invalid input, and authorization failures, because error paths often query the database or allocate response bodies differently from successful paths.

Keep large uploads and long exports in their own scenario if they have a different service target. Still run them concurrently with normal traffic. Otherwise the test misses the exact incident users notice: one export class occupies the pool while a simple settings page waits behind it.

Do not mock PostgreSQL, object storage, queues, or outbound services in the final capacity run. A mock is useful for isolating handler cost, but it removes the dependencies most likely to set capacity. Point the test at a staging stack with the same instance sizes, database settings, indexes, connection limits, and network path as production. Scrubbed production-shaped data is better than a thousand identical seed rows.

Avoid testing through a content delivery cache if the API is not normally cached. Conversely, keep the real cache in the path when production uses it. The purpose is not to make the backend look busy. The purpose is to reproduce the work a user request actually causes.

A runnable k6 test should encode the contract

Put thresholds and traffic stages in version control so a test run cannot become a screenshot interpreted after the fact. k6 treats thresholds as pass or fail criteria and exits nonzero when they fail, which makes the result suitable for a release check.

The following skeleton drives a mixed session at an arrival rate, checks response semantics, and places separate latency limits on ordinary reads and heavy exports. Replace the routes, payloads, and targets with values agreed for your product. Code inside the test is deliberately plain so a failed check maps to a user action.

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';

const businessErrors = new Rate('business_errors');

export const options = {
  scenarios: {
    expected_peak: {
      executor: 'ramping-arrival-rate',
      startRate: 5,
      timeUnit: '1s',
      preAllocatedVUs: 40,
      maxVUs: 200,
      stages: [
        { target: 10, duration: '5m' },
        { target: 10, duration: '30m' },
        { target: 20, duration: '10m' },
        { target: 20, duration: '30m' },
      ],
    },
  },
  thresholds: {
    'http_req_duration{name:project_list}': ['p(95)<300'],
    'http_req_duration{name:project_create}': ['p(95)<500'],
    'http_req_duration{name:export}': ['p(95)<2000'],
    http_req_failed: ['rate<0.01'],
    business_errors: ['rate<0.005'],
    dropped_iterations: ['count==0'],
  },
};

export function setup() {
  const response = http.post(`${__ENV.BASE_URL}/api/login`, JSON.stringify({
    email: __ENV.TEST_EMAIL,
    password: __ENV.TEST_PASSWORD,
  }), { headers: { 'Content-Type': 'application/json' } });
  check(response, { 'login succeeds': r => r.status === 200 });
  return { token: response.json('token') };
}

export default function (data) {
  const headers = { Authorization: `Bearer ${data.token}`, 'Content-Type': 'application/json' };
  const list = http.get(`${__ENV.BASE_URL}/api/projects?limit=25`, {
    headers,
    tags: { name: 'project_list' },
  });
  businessErrors.add(!check(list, {
    'list status is 200': r => r.status === 200,
    'list has items': r => Array.isArray(r.json('items')),
  }));

  if (Math.random() < 0.25) {
    const create = http.post(`${__ENV.BASE_URL}/api/projects`, JSON.stringify({
      name: `load-${__VU}-${__ITER}`,
    }), { headers, tags: { name: 'project_create' } });
    businessErrors.add(!check(create, { 'create status is 201': r => r.status === 201 }));
  }

  if (Math.random() < 0.03) {
    const runExport = http.post(`${__ENV.BASE_URL}/api/exports`, '{}', {
      headers,
      tags: { name: 'export' },
    });
    businessErrors.add(!check(runExport, { 'export accepted': r => r.status === 202 }));
  }

  sleep(Math.random() * 2 + 1);
}

The sample targets are starting points, not universal promises. Set p95 per route class from the delay a user can tolerate and the product's own requirements. Do not use one global 300 ms threshold to judge an asynchronous export and an autocomplete request.

Run the script from a machine that is not hosting the application. Confirm the generator has spare CPU and no dropped iterations. Save the exact commit, environment configuration, data snapshot identifier, command, and raw test output. Without those, a later comparison is mostly memory and optimism.

Calibrate the generator before trusting a long run. Point it at a tiny handler with no database work, raise the requested arrival rate above the planned test, and check that the generator sustains the rate without exhausting its own CPU, sockets, or network. When k6 adds virtual users or reports dropped iterations, the load machine may be the limit. Distribute generation across machines only when one machine cannot offer the required work, and keep their clocks synchronized so server and client graphs line up.

Give each run a quiet baseline. Stop migrations, data imports, and unrelated staging jobs unless those jobs also run during the real peak. Then schedule a second test with genuine background work enabled. The pair tells you both the API clean capacity and the operating capacity users will receive. If only the quiet run passes, the launch plan depends on a staging fiction.

Build a second script for a single user journey and run it with one virtual user before adding load. Inspect every response, created record, and cleanup action. This catches a bad token, a check that always returns true, or test data that collides after the first iteration. A capacity result is meaningless when the script is exercising an error page or repeatedly reading the same cached object.

p95 needs route-level context

Use p95 latency because averages conceal a slow minority, but never read p95 alone. At p95 of 800 ms, one request in twenty takes at least that long, which can make a multi-request page feel consistently slow. The percentile also becomes noisy on routes with very few samples, so report the request count beside it.

Record p50, p95, p99, maximum, throughput, and error rate for each named business action. p50 shows normal behavior, p95 is a practical service gate, and p99 exposes the tail without letting a single maximum dominate the conversation. Split results by status code. A fast 500 response must not improve the latency story.

Measure server-side handler duration as well as client-observed duration. The gap includes connection setup, proxies, network time, and response transfer. If client p95 rises while handler p95 stays flat, look outside the handler. If both rise and database wait time climbs, the request is probably queuing for a connection or query.

Warm-up and steady-state results must remain separate. Compilation is not occurring in a deployed Go binary, but cold caches, new database connections, lazy initialization, and autoscaling can distort the opening minutes. Users still experience cold behavior, so keep it as a distinct result rather than deleting it.

Averages remain useful for resource accounting. Total database time divided by calls can identify a query that is moderately slow and extremely frequent. It just cannot replace a percentile service target. The distinction the field often blurs is latency versus capacity: latency describes how long completed work took, while capacity describes how much offered work the service can sustain without growing queues or errors. Good latency at a low achieved request rate does not prove capacity.

Define failure before the run. I would fail a launch test when any critical route misses its p95 target, unexpected HTTP failures exceed the agreed rate, business checks fail, scheduled iterations drop, or a resource remains saturated. Passing four of five gates is a failed run with useful diagnostic data.

Connection waits expose hidden database queues

Instrument database/sql before the test, because application latency cannot tell you whether PostgreSQL is slow or the application is waiting to reach it. Go's DB.Stats() reports OpenConnections, InUse, Idle, WaitCount, and WaitDuration, along with closure counters. Export them to your metrics system every few seconds.

func recordDBStats(ctx context.Context, db *sql.DB, g GaugeSet) {
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            s := db.Stats()
            g.Set("db_open_connections", float64(s.OpenConnections))
            g.Set("db_in_use_connections", float64(s.InUse))
            g.Set("db_idle_connections", float64(s.Idle))
            g.Set("db_wait_count_total", float64(s.WaitCount))
            g.Set("db_wait_seconds_total", s.WaitDuration.Seconds())
        }
    }
}

Calculate the change in cumulative counters over the steady window. WaitCount increasing means requests had to wait for a free connection. The change in WaitDuration divided by the change in WaitCount gives average pool wait for that interval. Graph InUse against MaxOpenConnections; a flat line at the limit plus rising waits is pool saturation.

Do not respond by raising SetMaxOpenConns until the graph looks better. That popular fix moves the queue into PostgreSQL and can increase contention, memory use, and query latency. First identify why connections stay busy: slow queries, transactions held during network calls, rows read one at a time, or forgotten Rows.Close() calls. Then size the pool within the database's connection budget across every application replica and worker.

The Go documentation says a nonpositive SetMaxOpenConns value leaves the pool unlimited. Unlimited is a dangerous production default when several replicas can open connections at once. Set an explicit limit, set idle and lifetime behavior deliberately, and reserve database capacity for migrations, administration, and background work.

Track transaction duration separately. A handler may return in 200 ms while a deferred cleanup or leaked transaction holds a connection much longer. Pool metrics reveal the pressure, but traces or transaction timing reveal the owner.

Slow-query evidence must come from PostgreSQL

Enable pg_stat_statements in the test environment and take snapshots before and after each run. PostgreSQL's documentation describes it as tracking planning and execution statistics for normalized statements. Its view includes calls, rows, total and mean execution time, block activity, and temporary block activity. That evidence is far better than guessing from whichever query appeared in one trace.

Use a delta between snapshots because the view is cumulative. Reset it only in an isolated test database, since a reset destroys evidence for other work. This query finds statements that consumed the most execution time during a clean test window:

SELECT
  queryid,
  calls,
  round(total_exec_time::numeric, 1) AS total_ms,
  round(mean_exec_time::numeric, 2) AS mean_ms,
  rows,
  shared_blks_read,
  temp_blks_written
FROM pg_stat_statements
WHERE calls > 20
ORDER BY total_exec_time DESC
LIMIT 20;

Total time finds frequent queries that dominate database work. Mean time finds individually slow statements. Neither gives query p95 because pg_stat_statements aggregates calls; use tracing or duration histograms when tail latency for one statement matters. That is another routinely blurred distinction: a slow query can mean high average execution time, high tail time, or simply huge total cost from frequent calls. Each requires a different fix.

For the top statements, run EXPLAIN (ANALYZE, BUFFERS) against safe, representative parameters outside the timed load test. ANALYZE executes the statement, so wrap data-changing statements in a transaction that you roll back, or inspect them on a disposable copy. Look for estimates that differ sharply from actual rows, repeated loops, sequential scans over large selective tables, sorts spilling to disk, and many shared blocks read.

Generated code often creates an N+1 pattern that looks fine with seed data: fetch 25 projects, then issue one owner query and one count query per project. At ten requests per second, that one endpoint can create more than 500 database statements per second before any other route runs. The repair may be a join, a batched WHERE id = ANY($1), or a precomputed count. Raising the pool limit leaves the waste intact.

Capture lock waits too. A fast query in isolation can stall under concurrent updates to the same tenant, account, or sequence. When latency jumps only in the write scenario, inspect active waits and transaction boundaries instead of adding an index by reflex.

Memory must settle under a steady load

Judge memory by its shape over time, not by a single peak. A Go process that climbs during warm-up and then oscillates around a stable level is behaving differently from one whose post-garbage-collection baseline rises throughout a two-hour soak.

Record process resident memory, Go heap allocation, heap objects, goroutine count, garbage collection frequency, pause time, and allocation rate. Container memory matters because the operating system kills the process based on its limit, not on the Go heap alone. Compare memory after garbage collections at similar load. That removes much of the normal sawtooth and makes retention visible.

Expose the standard Go runtime metrics or a protected profiling endpoint in staging. Take a heap profile near the start and end of the soak, then compare retained allocation sites with go tool pprof. Take a goroutine profile as well. A growing goroutine count can reveal requests stuck on channels, response bodies that were never closed, or background work started without cancellation.

Do not set GOMEMLIMIT to the container limit exactly. The process also needs memory for goroutine stacks, executable mappings, database driver buffers, and other non-heap allocations. Leave headroom and prove it under the largest payloads. A test with tiny JSON bodies says little about an endpoint that reads a 20 MB upload into memory.

Force the awkward cases: maximum accepted request bodies, large query results, cancelled clients, timeouts, and repeated exports. Verify that memory returns after work completes. Also watch CPU, because heavy garbage collection can hold memory below its limit while destroying latency.

A practical pass condition combines a ceiling and a trend. Require resident memory to stay safely below the deployment limit at growth peak, then require the post-collection baseline and goroutine count to stop rising during the soak. There is no universal safe percentage. Choose headroom for the platform's restart behavior, traffic variance, and whether another replica can absorb a restart.

Failures must include wrong answers and overload behavior

Count transport failures, HTTP status failures, timeouts, panics, and wrong responses separately. http_req_failed catches failed HTTP requests according to k6's response callback, but a 200 response with an empty list, duplicate charge, or missing record is still a failure. That is why the sample emits business_errors from content checks.

Tag expected rejections such as invalid input or an intentional rate limit so they do not pollute the unexpected error rate. Then assert their contract: the status is correct, the body is bounded, and the rejection arrives quickly. A system under overload should not spend 30 seconds before returning 503.

Watch server logs for panic recovery, context deadline errors, connection acquisition delays, PostgreSQL serialization failures, and cancelled queries. Group errors by stable cause, not by full message, so identifiers do not create thousands of categories. Save representative traces for the first occurrence and for the high-latency tail.

Run one degradation test after the clean capacity test. Reduce available database connections, add controlled latency to one outbound dependency, or restart one application replica while traffic continues. Do this only in the isolated test environment. The goal is to confirm that timeouts, cancellation, and health checks contain the failure instead of allowing queues to consume every resource.

Configure the HTTP server explicitly. Go's net/http documentation states that zero or negative ReadTimeout, WriteTimeout, and IdleTimeout values can mean no timeout, depending on the field. Generated services often call http.ListenAndServe with defaults and never make the decision. A custom http.Server, request-scoped deadlines, and bounded body sizes prevent slow clients and stuck dependencies from holding resources forever.

Review correctness after the run. Count created records, verify idempotency where clients retried, check that background jobs finished once, and confirm no partial state survived failed requests. Load tests have uncovered more duplicate-work bugs in my projects than clever code review ever did.

The hiring decision comes from the first limit

You can launch without a backend engineer when the system passes the expected and growth tests repeatedly, the soak reaches a stable memory baseline, database queues remain controlled, and the team can explain the first stress failure. One lucky green run is not evidence. Run the same commit at least three times and investigate large variance.

Keep a compact result record for each run:

• Commit and environment, including replica sizes and database configuration.
• Dataset scale, traffic mix, arrival rates, and test duration.
• Route p95 and p99, achieved throughput, business failures, and dropped iterations.
• Peak CPU and memory, post-collection memory trend, and goroutine trend.
• Pool waits, top SQL by total time, lock waits, and the observed breaking point.

Hire or contract backend help before launch when nobody can explain rising pool waits, retained memory, lock contention, or inconsistent writes. Hire when the only person able to run the test cannot safely change the generated code. That is an ownership gap, not a request-per-second threshold.

A failing test does not automatically justify a full-time hire. A missing index, an N+1 query, or an unbounded export may be a contained repair. Repeated failures across transaction design, observability, cancellation, and deployment behavior point to ongoing engineering work. The distinction is whether you found one defect or discovered that nobody owns the system's behavior.

Koder.ai can generate and export a Go backend, deploy it, and preserve snapshots for rollback, but generation does not repeal capacity planning. Keep the load script and observability changes with the source so each meaningful backend change must pass the same contract.

Do not promise the business that 10,000 monthly users are safe. Promise a measured arrival rate, route mix, latency target, error budget, and resource envelope. When the product changes, change those inputs and run the test again.