Performance, Security, Resilience, and Production

Non-functional quality must be tied to realistic workloads, explicit objectives, sensitive-data boundaries, recoverable failure, and observable production behavior.

Performance workflow

flowchart LR
    A[Business periods and user roles] --> B[Transaction mix and arrival rate]
    B --> C[Data volume dependencies and cache state]
    C --> D[Baseline load stress spike soak volume tests]
    D --> E[Latency throughput errors saturation and recovery]
    E --> F{Objectives met under representative load}
    F -->|No| G[Find bottleneck fix and rerun same workload]
    F -->|Yes| H[Capacity and release evidence]

Do not begin with a single statement such as “10,000 users.” Define roles, geography, transaction mix, concurrency or arrival rate, read/write ratio, data history, think time, background jobs, external dependencies, warm-up, and cache state. Measure p50/p95/p99 latency, throughput, errors, saturation, CPU, memory, garbage collection, database behavior, connection pools, locks, cache hit rate, queue depth, Kafka lag, dependency calls, and recovery after load.

Security and privacy

Apply controls throughout the SDLC: privacy and authorization requirements, threat modeling, secure coding, SAST/SCA/secret/container/IaC scans, API and tenant-isolation testing, penetration testing, deployment checks, monitoring, incident response, and access review.

Critical negative tests include cross-user and cross-tenant identifiers, role escalation, token/session/logout behavior, upload/download authorization, leakage through exports/search/cache/jobs/reports/AI retrieval, prompt injection, and secrets or PII in repositories, logs, prompts, traces, and errors.

Resilience experiment

flowchart TD
    A[Define service SLO RTO RPO and steady state] --> B[Inject one controlled failure]
    B --> C[Observe routing retries breakers queues alerts and user impact]
    C --> D[Recover service]
    D --> E[Reconcile state and data]
    E --> F[Compare detection and recovery with objectives]
    F --> G[Increase blast radius gradually]

Useful experiments include terminating one service instance, slowing an external tax provider, making a banking sandbox unavailable, interrupting Kafka, removing a read replica, stopping a batch partway through, building queue backlog, and restoring from backup. For state-changing flows, prove idempotency and reconciliation after recovery.

CI/CD gates

• Pull request: build/lint, unit/component, changed contracts, security/license policy, skipped-test checks, human review.
• Integration/deployment: service integration, database compatibility, critical smoke, authorization and tenant negatives, observability.
• Release candidate: risk regression, critical E2E, data reconciliation, AI locked-set evaluation, performance, security, failure recovery, rollback or safe disable.

Validate discovered, executed, skipped, and quarantined test counts. A zero process exit code is insufficient by itself.

Production assurance

Use synthetic checks, SLOs and error budgets, traces and correlation IDs, payroll/payment reconciliation, Kafka and queue monitoring, ETL freshness, AI quality/latency/cost/drift metrics, canary rollout, rollback criteria, and incident reviews that produce new tests. Logs and traces must support diagnosis without creating a privacy leak.

Back to index