Phase 6: Valkey availability mirror — move read path off Postgres
Mitra-availability state (online flag, deactivated flag, per-mitra session count, heartbeat liveness) mirrored into Valkey so the customer beacon + pairing blast + dashboard counts no longer hit Postgres on the hot path. Postgres remains the durable source of truth; Valkey state is fully derivable via seedFromPostgres on startup + reconnect. Schema - mitras:online SET — mirror of is_online - mitras:deactivated SET — mirror of is_active=false - mitra:capacity:<id> STRING — active+pending_payment session count - mitra💓<id> STRING — ISO timestamp of last ping - availability:snapshot JSON — beacon cache, TTL 10s, cluster-shared Write paths (Postgres first, best-effort Valkey) - setOnline/setOffline mirror SADD/SREM + heartbeat SET/DEL - updateMitraStatus mirrors mitras:deactivated AND revokes auth_sessions on deactivate (bounds the "ghost online" window to access-token TTL) - heartbeat is Valkey-only on the hot path; the per-ping Postgres UPDATE on last_heartbeat_at is eliminated (was 1,200 ops/min at prod scale) - chat_session lifecycle (accept/end/reroute/extension/expiry) calls recomputeCapacityForMitra after each UPDATE — derive-from-truth avoids the bookkeeping risk of per-transition INCR/DECR Read paths (Valkey-first, Postgres fallback on Valkey error) - isMitraReachable: SISMEMBER mitras:online + heartbeat freshness - findAvailableMitras: SDIFF + pipelined GETs, filter by capacity + heartbeat - countAvailableMitrasFromCache: Valkey-driven, cached cluster-wide 10s TTL - dashboard online count: SCARD - Each reader wraps Valkey ops in try/catch → Postgres fallback on outage Heartbeat path on /api/mitra/status/heartbeat - resolveMitra preHandler replaced with heartbeatGuard: SISMEMBER on mitras:deactivated (~0 DB hits per ping). Falls back to full DB resolveMitra if Valkey is unreachable so a Valkey outage doesn't silently accept heartbeats from deactivated mitras. Three sweeps, env-configurable cadences - MITRA_AUTO_OFFLINE_SWEEP_SECONDS (30) — Valkey-driven stale detection - HEARTBEAT_MIRROR_INTERVAL_SECONDS (60) — batched UPSERT writes Valkey timestamps to Postgres last_heartbeat_at via UNNEST (1 statement per cycle, idempotent across instances) - VALKEY_ONLINE_MIRROR_SWEEP_SECONDS (300) — periodic reseed heals drift Startup - restoreActiveTimers → seedFromPostgres → bind listeners - onValkeyReady re-runs the seed on every reconnect (cold start + reseed on Valkey restart, no manual intervention) Failure semantics - Read fallback: every Valkey read wrapped, falls back to existing Postgres JOIN query — system stays correct during Valkey outage, performance degrades not breaks - Write best-effort: Postgres write commits before Valkey is touched; Valkey errors log + continue; reconciliation sweep heals drift - Auto-offline sweep aborts entirely on Valkey error (does NOT mass- offline via Postgres scan during Valkey hiccup) Tests - New: 32 integration tests in mitra-status.valkey-mirror.test.js covering seed, write-through, fallbacks, capacity lifecycle, auto-offline sweep, heartbeat mirror, deactivation flow, beacon cache - Updated: fixtures.js seeds Valkey alongside Postgres when isOnline=true - Updated: helpers/db.js resetDb also flushes test Valkey - Fixed 2 pre-existing session-timer flakes (string IDs failed uuid parse; vi.advanceTimersByTimeAsync raced real Postgres I/O) - All 124/124 backend tests pass (was 90/92) Docs - requirement/valkey-online-mirror-plan.md — canonical plan - requirement/valkey-online-mirror-testing.md — manual E2E checklist - requirement/deployment.md — infra + Valkey persistence guidance for prod (Memorystore Standard tier recommended; migration from self-hosted Valkey is zero-downtime via reseed-from-Postgres) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,8 @@
|
||||
import { authenticate } from '../../plugins/auth.js'
|
||||
import { getMitraById } from '../../services/mitra.service.js'
|
||||
import * as mitraStatusService from '../../services/mitra-status.service.js'
|
||||
import * as valkey from '../../plugins/valkey.js'
|
||||
import { VK_MITRAS_DEACTIVATED } from '../../services/mitra-status.service.js'
|
||||
import { UserType } from '../../constants.js'
|
||||
|
||||
export const mitraStatusRoutes = async (app) => {
|
||||
@@ -27,6 +29,32 @@ export const mitraStatusRoutes = async (app) => {
|
||||
request.mitra = mitra
|
||||
}
|
||||
|
||||
// Lightweight heartbeat guard: no Postgres SELECT in the hot path. Checks
|
||||
// `mitras:deactivated` in Valkey (maintained on every updateMitraStatus) and
|
||||
// falls back to the full resolveMitra/DB check if Valkey is unreachable so a
|
||||
// Valkey outage doesn't accept heartbeats from deactivated mitras silently.
|
||||
const heartbeatGuard = async (request, reply) => {
|
||||
if (request.auth?.userType !== UserType.MITRA) {
|
||||
return reply.code(403).send({
|
||||
success: false,
|
||||
error: { code: 'FORBIDDEN', message: 'Mitra account required' },
|
||||
})
|
||||
}
|
||||
try {
|
||||
const deactivated = await valkey.sismember(VK_MITRAS_DEACTIVATED, request.auth.userId)
|
||||
if (deactivated) {
|
||||
return reply.code(403).send({
|
||||
success: false,
|
||||
error: { code: 'ACCOUNT_INACTIVE', message: 'Account is inactive' },
|
||||
})
|
||||
}
|
||||
return
|
||||
} catch (err) {
|
||||
console.warn('[heartbeat] valkey check failed, falling back to DB:', err.message)
|
||||
return resolveMitra(request, reply)
|
||||
}
|
||||
}
|
||||
|
||||
app.post('/online', { preHandler: [authenticate, resolveMitra] }, async (request, reply) => {
|
||||
await mitraStatusService.setOnline(request.mitra.id)
|
||||
return reply.send({ success: true, data: { is_online: true } })
|
||||
@@ -37,8 +65,8 @@ export const mitraStatusRoutes = async (app) => {
|
||||
return reply.send({ success: true, data: { is_online: false } })
|
||||
})
|
||||
|
||||
app.post('/heartbeat', { preHandler: [authenticate, resolveMitra] }, async (request, reply) => {
|
||||
await mitraStatusService.heartbeat(request.mitra.id)
|
||||
app.post('/heartbeat', { preHandler: [authenticate, heartbeatGuard] }, async (request, reply) => {
|
||||
await mitraStatusService.heartbeat(request.auth.userId)
|
||||
return reply.send({ success: true })
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user