Architecture Overview
Last updated: March 19, 202610 min readAdvancedDeveloper
This document provides a technical overview of the Arqam360 consent data pipeline architecture.
Tags & Pixels
Consent records flow through a BullMQ-based event pipeline: consent record saved → enqueued to 'consent-events' queue → eventPipelineWorker evaluates routing rules → fans out to 'integration-forward' queue per platform → integrationForwardWorker delivers with retry.
Data Flow
- Widget collects consent → POST /api/v1/consent/record
- Backend saves ConsentRecord → enqueues to consent-events
- Policy engine evaluates RoutingRules → determines target platforms
- Platform connectors transform and forward to each destination
- RoutingDecision audit trail logged for every forwarding decision
BullMQ Queues
- consent-events — Primary consent event processing
- integration-forward — Per-platform forwarding with retry
- webhook-dispatch — Reliable webhook delivery
Workers
- eventPipelineWorker — Routes consent events through policy engine
- integrationForwardWorker — Delivers to platforms with retry and circuit breaker
- webhookWorker — Delivers webhooks with 3x retry
- dataEventWorker — Processes server-side events
- segmentEvalWorker — Evaluates segment membership
- audienceSyncWorker — Syncs audiences to ad platforms
Resilience Patterns
The system uses Redis-backed circuit breakers (5 failure threshold, 60s reset), BullMQ retry with exponential backoff, and graceful degradation to sync HTTP calls if Redis is unavailable.
The event pipeline requires Redis (REDIS_URL env var). Without it, consent forwarding falls back to legacy synchronous HTTP calls with no retry or routing rules.
Was this article helpful?