docs: merge ARCHITECTURE.md files into single consolidated reference
Some checks failed
Build and Push Docker Image / build (push) Successful in 43s
Docs Check / Markdown lint (push) Successful in 47s
CI / Security audit (push) Has been cancelled
CI / Tests & coverage (push) Has been cancelled
Docs Check / Mermaid diagram parse check (push) Has been cancelled
Some checks failed
Build and Push Docker Image / build (push) Successful in 43s
Docs Check / Markdown lint (push) Successful in 47s
CI / Security audit (push) Has been cancelled
CI / Tests & coverage (push) Has been cancelled
Docs Check / Mermaid diagram parse check (push) Has been cancelled
- Combine root ARCHITECTURE.md (webhook/smart-polling focused) with docs/ARCHITECTURE.md (deep-dive) into one authoritative document - Structured into 11 sections: Introduction, High-Level Architecture, Pluggable Architecture Layers (PDCA + PALDRA), Webhook System, Data Flow, Caching & Smart Polling, Key Subsystems, Directory Structure, Configuration, Security Model, Technology Stack - Add full-system Mermaid flowchart, webhook sequence diagram, polling cycle sequence diagram, UI state machine, download matching flowchart - Document all cache keys, NormalizedDownload schema, DownloadClientRegistry and arrRetrieverRegistry APIs, webhook event classification table, complete security model with auth/webhook/headers subsections - Remove all development-phase references and internal process language - Remove docs/ARCHITECTURE.md (content consolidated into root file)
This commit is contained in:
880
ARCHITECTURE.md
880
ARCHITECTURE.md
@@ -1,35 +1,120 @@
|
|||||||
# sofarr — Architecture Reference
|
# sofarr — Architecture
|
||||||
|
|
||||||
> Concise top-level architecture guide. For the full deep-dive (API reference, matching pipeline, deployment) see [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
|
Comprehensive technical reference for the **sofarr** application: a personal media download dashboard that aggregates download activity from SABnzbd, Sonarr, Radarr, qBittorrent, Transmission, and rTorrent, filters results by Emby/Jellyfin user identity, and presents a real-time personalised dashboard.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Overview
|
## Table of Contents
|
||||||
|
|
||||||
sofarr is a **Node.js/Express** single-page application. It aggregates download activity from multiple media automation services, filters results by Emby user identity, and presents a real-time personalised dashboard.
|
1. [Introduction](#1-introduction)
|
||||||
|
2. [High-Level Architecture](#2-high-level-architecture)
|
||||||
|
3. [Pluggable Architecture Layers](#3-pluggable-architecture-layers)
|
||||||
|
4. [Webhook System](#4-webhook-system)
|
||||||
|
5. [Data Flow and Real-time Updates](#5-data-flow-and-real-time-updates)
|
||||||
|
6. [Caching and Smart Polling](#6-caching-and-smart-polling)
|
||||||
|
7. [Key Subsystems](#7-key-subsystems)
|
||||||
|
8. [Directory Structure](#8-directory-structure)
|
||||||
|
9. [Configuration and Environment Variables](#9-configuration-and-environment-variables)
|
||||||
|
10. [Security Model](#10-security-model)
|
||||||
|
11. [Technology Stack](#11-technology-stack)
|
||||||
|
|
||||||
Three pluggable layers form the core:
|
---
|
||||||
|
|
||||||
|
## 1. Introduction
|
||||||
|
|
||||||
|
sofarr is a **Node.js/Express single-page application** that provides a personalised view of media downloads. It:
|
||||||
|
|
||||||
|
1. **Authenticates** users against an Emby/Jellyfin media server.
|
||||||
|
2. **Aggregates** download data from multiple *arr service instances and download clients.
|
||||||
|
3. **Filters** downloads per user — each user only sees media tagged with their username in Sonarr/Radarr.
|
||||||
|
4. **Presents** a real-time dashboard with progress, speeds, cover art, and status, updated either via background polling or instant webhook push from Sonarr/Radarr.
|
||||||
|
|
||||||
|
Admin users can view all users' downloads, see server status, cache statistics, poll timings, and perform blocklist-and-search operations.
|
||||||
|
|
||||||
|
Three pluggable layers form the architectural core:
|
||||||
|
|
||||||
| Layer | Name | Location |
|
| Layer | Name | Location |
|
||||||
|-------|------|----------|
|
|-------|------|----------|
|
||||||
| Download client abstraction | **PDCA** — Pluggable Download Client Architecture | `server/clients/` + `server/utils/downloadClients.js` |
|
| Download client abstraction | **PDCA** — Pluggable Download Client Architecture | `server/clients/` + `server/utils/downloadClients.js` |
|
||||||
| *arr data retrieval | **PALDRA** — Pluggable *Arr Library Data Retrieval Architecture | `server/utils/arrRetrievers.js` |
|
| *arr data retrieval | **PALDRA** — Pluggable *Arr Library Data Retrieval Architecture | `server/utils/arrRetrievers.js` |
|
||||||
| Real-time push | **Webhook receiver** | `server/routes/webhook.js` |
|
| Real-time push | **Webhook Receiver** | `server/routes/webhook.js` |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Request / Data Flow
|
## 2. High-Level Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TB
|
||||||
|
subgraph Browser["Browser (SPA — public/)"]
|
||||||
|
login["Login Form"]
|
||||||
|
dash["Dashboard Cards"]
|
||||||
|
status["Status Panel\n(Admin only)"]
|
||||||
|
history["History Tab"]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Server["Express Server (:3001)"]
|
||||||
|
direction TB
|
||||||
|
mw["Middleware\nHelmet · rate-limit · cookie-parser · verifyCsrf"]
|
||||||
|
auth_r["Auth Routes\n/api/auth"]
|
||||||
|
dash_r["Dashboard Routes\n/api/dashboard (SSE /stream)"]
|
||||||
|
wh_r["Webhook Routes\n/api/webhook/sonarr|radarr"]
|
||||||
|
hist_r["History Routes\n/api/history"]
|
||||||
|
proxy_r["Proxy Routes\n/api/sonarr · /api/radarr\n/api/sabnzbd · /api/emby"]
|
||||||
|
|
||||||
|
subgraph Core["Core Utilities"]
|
||||||
|
poller["Poller\n(smart background polling)"]
|
||||||
|
cache["MemoryCache\n(poll:* + webhook metrics)"]
|
||||||
|
pdca["PDCA Registry\n(download clients)"]
|
||||||
|
paldra["PALDRA Registry\n(arr retrievers)"]
|
||||||
|
tokenstore["TokenStore\n(tokens.json)"]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Ext["External Services"]
|
||||||
|
sab["SABnzbd"]
|
||||||
|
sonarr["Sonarr"]
|
||||||
|
radarr["Radarr"]
|
||||||
|
qbt["qBittorrent"]
|
||||||
|
rtorrent["rTorrent"]
|
||||||
|
transmission["Transmission"]
|
||||||
|
emby["Emby / Jellyfin"]
|
||||||
|
end
|
||||||
|
|
||||||
|
login -->|"POST /api/auth/login"| auth_r
|
||||||
|
dash -->|"GET /api/dashboard/stream (SSE)"| dash_r
|
||||||
|
status -->|"GET /api/dashboard/status"| dash_r
|
||||||
|
history -->|"GET /api/history/recent"| hist_r
|
||||||
|
|
||||||
|
auth_r --> tokenstore
|
||||||
|
auth_r -->|"authenticate"| emby
|
||||||
|
|
||||||
|
dash_r --> cache
|
||||||
|
dash_r --> poller
|
||||||
|
wh_r --> cache
|
||||||
|
wh_r --> paldra
|
||||||
|
hist_r --> cache
|
||||||
|
proxy_r -->|"proxy"| sonarr & radarr & sab & emby
|
||||||
|
|
||||||
|
poller --> pdca & paldra
|
||||||
|
poller --> cache
|
||||||
|
pdca -->|"HTTP/API"| sab & qbt & rtorrent & transmission
|
||||||
|
paldra -->|"HTTP/API"| sonarr & radarr
|
||||||
|
|
||||||
|
sonarr & radarr -->|"POST /api/webhook/*"| wh_r
|
||||||
|
```
|
||||||
|
|
||||||
|
### Request routing summary
|
||||||
|
|
||||||
```
|
```
|
||||||
Browser (SPA)
|
Browser (SPA)
|
||||||
│ POST /api/auth/login → Auth routes → Emby verify → set httpOnly cookie
|
│ POST /api/auth/login → Auth routes → Emby verify → set httpOnly cookie
|
||||||
│ GET /api/dashboard/stream → SSE stream → poller cache → matched downloads
|
│ GET /api/dashboard/stream → SSE stream → cache → matched downloads
|
||||||
│ POST /api/webhook/* ← Sonarr/Radarr push events
|
│ POST /api/webhook/* ← Sonarr/Radarr push events
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
Express Server (:3001)
|
Express Server (:3001)
|
||||||
├── Helmet (CSP nonce, HSTS, X-Frame-Options, …)
|
├── Helmet (CSP nonce, HSTS, X-Frame-Options, …)
|
||||||
├── express-rate-limit (300/15 min general; 60/1 min webhook)
|
├── express-rate-limit (300/15 min general; 60/1 min webhook; 10 fails/15 min login)
|
||||||
├── cookie-parser (HMAC-signed session cookie)
|
├── cookie-parser (HMAC-signed session cookie)
|
||||||
├── verifyCsrf (double-submit cookie, all state-changing /api routes except auth + webhook)
|
├── verifyCsrf (double-submit cookie, all state-changing /api routes except auth + webhook)
|
||||||
│
|
│
|
||||||
@@ -58,46 +143,177 @@ Background:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Pluggable Download Client Architecture (PDCA)
|
## 3. Pluggable Architecture Layers
|
||||||
|
|
||||||
All download clients extend `DownloadClient` (abstract base in `server/clients/DownloadClient.js`):
|
### 3.1 Pluggable Download Client Architecture (PDCA)
|
||||||
|
|
||||||
|
#### Overview
|
||||||
|
|
||||||
|
The PDCA provides a unified, extensible interface for all download clients. This abstraction layer enables:
|
||||||
|
|
||||||
|
- **Client-agnostic polling** — the poller contains no client-specific logic.
|
||||||
|
- **Easy extension** — add a new client by implementing one interface.
|
||||||
|
- **Consistent normalisation** — all clients return standardised download objects.
|
||||||
|
- **Centralised configuration** — a single registry manages all instances.
|
||||||
|
- **Error isolation** — individual client failures do not affect other clients.
|
||||||
|
|
||||||
|
#### Abstract Base Class
|
||||||
|
|
||||||
|
All download clients extend `DownloadClient` (`server/clients/DownloadClient.js`):
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
class DownloadClient {
|
||||||
|
constructor(instanceConfig)
|
||||||
|
getClientType(): string
|
||||||
|
getInstanceId(): string
|
||||||
|
async testConnection(): Promise<boolean>
|
||||||
|
async getActiveDownloads(): Promise<NormalizedDownload[]>
|
||||||
|
async getClientStatus(): Promise<Object|null> // optional
|
||||||
|
normalizeDownload(download): NormalizedDownload
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Client Implementations
|
||||||
|
|
||||||
```
|
```
|
||||||
DownloadClient (abstract)
|
DownloadClient (abstract)
|
||||||
├── SABnzbdClient — REST API, API key auth
|
├── SABnzbdClient — REST API, API key auth; handles queue + history; normalises time/size units
|
||||||
├── QBittorrentClient — Sync API (incremental deltas), cookie auth, fallback to /torrents/info
|
├── QBittorrentClient — Sync API (incremental deltas), cookie auth, fallback to /torrents/info
|
||||||
├── TransmissionClient — JSON-RPC, session-ID management
|
├── TransmissionClient — JSON-RPC, session-ID management
|
||||||
└── RTorrentClient — XML-RPC, HTTP Basic Auth
|
└── RTorrentClient — XML-RPC (xmlrpc 1.3.2), HTTP Basic Auth; maps rTorrent states to normalised statuses
|
||||||
```
|
```
|
||||||
|
|
||||||
`DownloadClientRegistry` (`server/utils/downloadClients.js`) initialises all configured clients from `*_INSTANCES` env vars, fetches from all in parallel, and returns a `{ sabnzbd, qbittorrent, transmission, rtorrent }` map. Individual client failures are isolated.
|
#### Normalised Download Schema
|
||||||
|
|
||||||
**Adding a new client:** extend `DownloadClient`, implement `getActiveDownloads()` returning `NormalizedDownload[]`, register in the registry factory.
|
Every client returns objects conforming to this schema:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
interface NormalizedDownload {
|
||||||
|
id: string // Client-specific unique ID
|
||||||
|
title: string // Download title/name
|
||||||
|
type: 'usenet' | 'torrent' // Download type
|
||||||
|
client: string // Client identifier ('sabnzbd', 'qbittorrent', etc.)
|
||||||
|
instanceId: string // Instance identifier
|
||||||
|
instanceName: string // Instance display name
|
||||||
|
status: string // Normalised status (Downloading, Seeding, etc.)
|
||||||
|
progress: number // Progress percentage (0–100)
|
||||||
|
size: number // Total size in bytes
|
||||||
|
downloaded: number // Downloaded bytes
|
||||||
|
speed: number // Current speed in bytes/sec
|
||||||
|
eta: number | null // ETA in seconds, null if unknown
|
||||||
|
category?: string // Download category (optional)
|
||||||
|
tags?: string[] // Download tags (optional)
|
||||||
|
savePath?: string // Save path (optional)
|
||||||
|
addedOn?: string // Added timestamp (optional)
|
||||||
|
arrQueueId?: number // Sonarr/Radarr queue ID (optional)
|
||||||
|
arrType?: 'series' | 'movie' // Sonarr/Radarr type (optional)
|
||||||
|
raw?: any // Original client response (escape hatch)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Registry (`server/utils/downloadClients.js`)
|
||||||
|
|
||||||
|
`DownloadClientRegistry` manages all instances:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
class DownloadClientRegistry {
|
||||||
|
async initialize() // Create clients from config
|
||||||
|
getAllClients(): DownloadClient[]
|
||||||
|
getClient(instanceId): DownloadClient
|
||||||
|
getClientsByType(type): DownloadClient[]
|
||||||
|
async getAllDownloads(): NormalizedDownload[] // Fetch from all clients in parallel
|
||||||
|
async testAllConnections(): Promise<ConnectionTestResult[]>
|
||||||
|
async getAllClientStatuses(): Promise<ClientStatus[]>
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Configuration-driven:** reads from `*_INSTANCES` environment variables (JSON array format) with fallback to legacy `*_URL` / `*_API_KEY` / `*_USERNAME` / `*_PASSWORD` variables.
|
||||||
|
|
||||||
|
#### qBittorrent Sync API Details
|
||||||
|
|
||||||
|
Each `QBittorrentClient` instance maintains:
|
||||||
|
|
||||||
|
- **`lastRid`** — response ID from the previous `sync/maindata` call (starts at `0`).
|
||||||
|
- **`torrentMap`** — `Map<hash, torrent>` holding the complete state for every known torrent.
|
||||||
|
- **`fallbackThisCycle`** — boolean tracking whether this poll cycle has already fallen back to the legacy endpoint.
|
||||||
|
|
||||||
|
Per-cycle flow:
|
||||||
|
1. Attempt `GET /api/v2/sync/maindata?rid={lastRid}`.
|
||||||
|
2. If `full_update` is `true`, rebuild `torrentMap` from scratch.
|
||||||
|
3. Otherwise merge delta fields into existing entries; remove `torrents_removed` hashes.
|
||||||
|
4. On Sync API failure, fall back **once per cycle** to `GET /api/v2/torrents/info`.
|
||||||
|
5. If the fallback also fails, return an empty array for this cycle and log the error.
|
||||||
|
|
||||||
|
The rest of the application (poller, dashboard) receives data in the same format regardless of which path was taken.
|
||||||
|
|
||||||
|
#### Adding a New Download Client
|
||||||
|
|
||||||
|
1. Create `server/clients/MyClient.js` extending `DownloadClient`.
|
||||||
|
2. Implement `getActiveDownloads()` returning `NormalizedDownload[]`.
|
||||||
|
3. Register the class in the registry factory inside `server/utils/downloadClients.js`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Pluggable *Arr Retrieval Architecture (PALDRA)
|
### 3.2 Pluggable *arr Retrieval Layer (PALDRA)
|
||||||
|
|
||||||
`server/utils/arrRetrievers.js` provides `arrRetrieverRegistry` which:
|
#### Overview
|
||||||
- Initialises one retriever per configured Sonarr/Radarr instance
|
|
||||||
- Exposes `getQueuesByType()`, `getHistoryByType()`, `getTagsByType()` — returning results keyed by `sonarr` / `radarr`
|
|
||||||
- Results carry `{ instance: instanceId, data: … }` so callers can look up instance credentials
|
|
||||||
|
|
||||||
The poller and webhook processor both use the same registry, ensuring consistency.
|
`server/utils/arrRetrievers.js` exports `arrRetrieverRegistry`, a singleton that manages one `PollingSonarrRetriever` or `PollingRadarrRetriever` per configured instance. It provides a uniform interface for fetching queue, history, and tag data, keyed by service type.
|
||||||
|
|
||||||
|
The registry is used by both the background poller and the webhook processor, guaranteeing consistent data shapes across both update paths.
|
||||||
|
|
||||||
|
#### Registry API
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
arrRetrieverRegistry = {
|
||||||
|
async initialize() // idempotent; reads config once
|
||||||
|
getAllRetrievers(): ArrRetriever[]
|
||||||
|
getRetriever(instanceId): ArrRetriever | null
|
||||||
|
getRetrieversByType(type): ArrRetriever[] // 'sonarr' | 'radarr'
|
||||||
|
|
||||||
|
// Typed fetch methods — all return { sonarr: [...], radarr: [...] }
|
||||||
|
async getQueuesByType(): Promise<{ sonarr, radarr }>
|
||||||
|
async getHistoryByType(options?): Promise<{ sonarr, radarr }>
|
||||||
|
async getTagsByType(): Promise<{ sonarr, radarr }>
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each result element is `{ instance: instanceId, data: <arr API response> }`, allowing callers to look up instance credentials from `config.js`.
|
||||||
|
|
||||||
|
#### Retriever API Calls
|
||||||
|
|
||||||
|
| Task | Endpoint | Key Parameters |
|
||||||
|
|------|----------|----------------|
|
||||||
|
| Sonarr tags | `GET /api/v3/tag` | — |
|
||||||
|
| Sonarr queue | `GET /api/v3/queue` | `includeSeries=true`, `includeEpisode=true` |
|
||||||
|
| Sonarr history | `GET /api/v3/history` | `pageSize=10`, `includeEpisode=true` |
|
||||||
|
| Radarr tags | `GET /api/v3/tag` | — |
|
||||||
|
| Radarr queue | `GET /api/v3/queue` | `includeMovie=true` |
|
||||||
|
| Radarr history | `GET /api/v3/history` | `pageSize=10` |
|
||||||
|
|
||||||
|
All fetches across all instances run in parallel via `Promise.allSettled`, so a single failing instance does not block others.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Webhook Flow (Phase 1–5.1)
|
## 4. Webhook System
|
||||||
|
|
||||||
|
### 4.1 Webhook Receiver
|
||||||
|
|
||||||
|
sofarr exposes two webhook endpoints that Sonarr and Radarr can be configured to call on every automation event:
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/webhook/sonarr
|
||||||
|
POST /api/webhook/radarr
|
||||||
|
```
|
||||||
|
|
||||||
|
Both endpoints share identical processing logic:
|
||||||
|
|
||||||
```
|
```
|
||||||
Sonarr/Radarr
|
Sonarr/Radarr
|
||||||
POST /api/webhook/sonarr (X-Sofarr-Webhook-Secret: <secret>)
|
POST /api/webhook/sonarr
|
||||||
{
|
Headers: X-Sofarr-Webhook-Secret: <secret>
|
||||||
"eventType": "Grab",
|
Body: { "eventType": "Grab", "instanceName": "Main Sonarr",
|
||||||
"instanceName": "Main Sonarr",
|
"date": "2026-05-19T10:00:00.000Z", … }
|
||||||
"date": "2026-05-19T10:00:00.000Z",
|
|
||||||
…
|
|
||||||
}
|
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
webhookLimiter (60 req/min/IP)
|
webhookLimiter (60 req/min/IP)
|
||||||
@@ -115,19 +331,255 @@ Sonarr/Radarr
|
|||||||
cache.updateWebhookMetrics(instance.url) ← activates smart polling skip
|
cache.updateWebhookMetrics(instance.url) ← activates smart polling skip
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
processWebhookEvent('sonarr', 'Grab') [fire-and-forget]
|
200 { received: true } ← response sent immediately
|
||||||
├── classify: Grab → QUEUE_EVENT
|
|
||||||
├── arrRetrieverRegistry.getQueuesByType()
|
|
||||||
├── cache.set('poll:sonarr-queue', …, CACHE_TTL)
|
|
||||||
└── pollAllServices() → pollSubscribers.forEach(cb) → SSE push
|
|
||||||
│
|
│
|
||||||
▼
|
▼ (fire-and-forget)
|
||||||
200 { received: true } (returned immediately, before fire-and-forget completes)
|
processWebhookEvent(serviceType, eventType)
|
||||||
|
├── classify: QUEUE_EVENT or HISTORY_EVENT
|
||||||
|
├── arrRetrieverRegistry.getQueuesByType() / getHistoryByType()
|
||||||
|
├── cache.set('poll:sonarr-queue' | 'poll:sonarr-history', …, CACHE_TTL)
|
||||||
|
└── pollAllServices() → pollSubscribers.forEach(cb) → SSE push
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The 200 response is sent **before** the background cache refresh completes, ensuring Sonarr/Radarr never have to wait for sofarr's downstream API calls.
|
||||||
|
|
||||||
|
#### Event Classification
|
||||||
|
|
||||||
|
| Event type | Classification | Cache keys refreshed |
|
||||||
|
|------------|---------------|---------------------|
|
||||||
|
| `Grab`, `Download`, `DownloadFailed`, `ManualInteractionRequired` | `QUEUE_EVENT` | `poll:{type}-queue` |
|
||||||
|
| `DownloadFolderImported`, `ImportFailed`, `EpisodeFileRenamed`, `MovieFileRenamed`, `EpisodeFileRenamedBySeries` | `HISTORY_EVENT` | `poll:{type}-history` |
|
||||||
|
| `Test`, `Rename`, `SeriesAdd`, `SeriesDelete`, `MovieAdd`, `MovieDelete`, `MovieFileDelete`, `Health`, `ApplicationUpdate`, `HealthRestored` | Informational — no refresh | — |
|
||||||
|
|
||||||
|
#### Accepted Event Types
|
||||||
|
|
||||||
|
The full allowlist enforced by `validatePayload()`:
|
||||||
|
|
||||||
|
```
|
||||||
|
Test · Grab · Download · DownloadFailed · ManualInteractionRequired
|
||||||
|
DownloadFolderImported · ImportFailed
|
||||||
|
EpisodeFileRenamed · MovieFileRenamed · EpisodeFileRenamedBySeries
|
||||||
|
Rename · SeriesAdd · SeriesDelete · MovieAdd · MovieDelete · MovieFileDelete
|
||||||
|
Health · ApplicationUpdate · HealthRestored
|
||||||
|
```
|
||||||
|
|
||||||
|
Any `eventType` not in this set is rejected with `400 Bad Request`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Smart Polling Optimization (Phase 5)
|
### 4.2 Real-time Cache and SSE Integration
|
||||||
|
|
||||||
|
When a webhook event is classified as a `QUEUE_EVENT` or `HISTORY_EVENT`:
|
||||||
|
|
||||||
|
1. `arrRetrieverRegistry` fetches fresh data from the relevant *arr instances (in parallel, via PALDRA).
|
||||||
|
2. The result is written directly into the shared `MemoryCache` under the same `poll:*` key the poller uses — ensuring both paths produce identical cache shapes.
|
||||||
|
3. `pollAllServices()` is called, which iterates `pollSubscribers` and pushes the updated payload to every open SSE connection immediately.
|
||||||
|
|
||||||
|
The dashboard therefore receives fresh data within the round-trip time of the *arr API call, without waiting for the next poll cycle.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.3 Notification Management API
|
||||||
|
|
||||||
|
The `sonarr.js` and `radarr.js` route modules expose endpoints (under `/api/sonarr` and `/api/radarr` respectively, behind `requireAuth` + `verifyCsrf`) for one-click webhook configuration. These proxy to the *arr notification API to create, update, or remove the sofarr webhook connection in Sonarr/Radarr, using `SOFARR_BASE_URL` to construct the target URL.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Data Flow and Real-time Updates
|
||||||
|
|
||||||
|
### 5.1 Polling Cycle (background path)
|
||||||
|
|
||||||
|
Every `POLL_INTERVAL` ms the poller fetches all services in parallel:
|
||||||
|
|
||||||
|
| Task | API | Key parameters |
|
||||||
|
|------|-----|----------------|
|
||||||
|
| SABnzbd Queue | `GET /api?mode=queue` | `output=json` |
|
||||||
|
| SABnzbd History | `GET /api?mode=history` | `limit=10` |
|
||||||
|
| Sonarr Tags | `GET /api/v3/tag` | — |
|
||||||
|
| Sonarr Queue | `GET /api/v3/queue` | `includeSeries=true`, `includeEpisode=true` |
|
||||||
|
| Sonarr History | `GET /api/v3/history` | `pageSize=10`, `includeEpisode=true` |
|
||||||
|
| Radarr Tags | `GET /api/v3/tag` | — |
|
||||||
|
| Radarr Queue | `GET /api/v3/queue` | `includeMovie=true` |
|
||||||
|
| Radarr History | `GET /api/v3/history` | `pageSize=10` |
|
||||||
|
| qBittorrent | `GET /api/v2/sync/maindata?rid=N` | Fallback: `GET /api/v2/torrents/info` |
|
||||||
|
|
||||||
|
Results are stored in `MemoryCache` under `poll:*` keys with TTL `POLL_INTERVAL × 3`. Per-task timings are recorded in `lastPollTimings` for the admin status panel.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant Entry as index.js
|
||||||
|
participant Poller
|
||||||
|
participant PDCA as PDCA Registry
|
||||||
|
participant PALDRA as PALDRA Registry
|
||||||
|
participant Cache as MemoryCache
|
||||||
|
participant SSE as SSE Subscribers
|
||||||
|
|
||||||
|
Entry->>Poller: startPoller()
|
||||||
|
loop Every POLL_INTERVAL ms
|
||||||
|
Poller->>Poller: polling flag check (skip if concurrent)
|
||||||
|
Poller->>PDCA: getDownloadsByClientType()
|
||||||
|
Poller->>PALDRA: getQueuesByType() / getHistoryByType() / getTagsByType()
|
||||||
|
PDCA-->>Poller: { sabnzbd, qbittorrent, rtorrent, transmission }
|
||||||
|
PALDRA-->>Poller: { sonarr: [...], radarr: [...] }
|
||||||
|
Poller->>Cache: set poll:* keys (TTL = POLL_INTERVAL × 3)
|
||||||
|
Poller->>SSE: notify all subscribers → push data: frame
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Webhook Path (real-time update)
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant Arr as Sonarr/Radarr
|
||||||
|
participant WH as /api/webhook/sonarr
|
||||||
|
participant Cache as MemoryCache
|
||||||
|
participant PALDRA as PALDRA Registry
|
||||||
|
participant SSE as SSE Subscribers
|
||||||
|
|
||||||
|
Arr->>WH: POST /api/webhook/sonarr { eventType, instanceName, date }
|
||||||
|
WH->>WH: validateSecret + validatePayload + isReplay
|
||||||
|
WH->>Cache: updateWebhookMetrics(instance.url)
|
||||||
|
WH-->>Arr: 200 { received: true }
|
||||||
|
Note over WH: fire-and-forget begins
|
||||||
|
WH->>PALDRA: getQueuesByType() or getHistoryByType()
|
||||||
|
PALDRA-->>WH: fresh arr data
|
||||||
|
WH->>Cache: set poll:sonarr-queue / poll:sonarr-history
|
||||||
|
WH->>SSE: pollAllServices() → push data: frame to all clients
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.3 SSE Stream
|
||||||
|
|
||||||
|
When a browser opens `GET /api/dashboard/stream`:
|
||||||
|
|
||||||
|
1. Server sets `Content-Type: text/event-stream`, disables buffering (`X-Accel-Buffering: no`).
|
||||||
|
2. Immediately builds and sends the first payload (same matching logic as `/user-downloads`).
|
||||||
|
3. Registers a callback with the poller's `onPollComplete` subscriber set.
|
||||||
|
4. After every subsequent poll cycle (or webhook-triggered broadcast), the callback fires, rebuilds the payload, and writes a `data:` SSE frame.
|
||||||
|
5. A 25-second heartbeat comment (`: heartbeat`) keeps the connection alive through proxies.
|
||||||
|
6. On client disconnect: deregisters callback, stops heartbeat, removes from `activeClients` map.
|
||||||
|
|
||||||
|
The browser's native `EventSource` API handles reconnection automatically on network interruption.
|
||||||
|
|
||||||
|
### 5.4 Download Matching Pipeline
|
||||||
|
|
||||||
|
For each connected user the server:
|
||||||
|
|
||||||
|
1. Reads all `poll:*` keys from `MemoryCache`.
|
||||||
|
2. Builds `seriesMap`, `moviesMap`, `sonarrTagMap`, and `radarrTagMap` from embedded objects in queue records.
|
||||||
|
3. For each SABnzbd/qBittorrent download item, attempts matches in priority order: Sonarr queue → Radarr queue → Sonarr history → Radarr history.
|
||||||
|
4. Title matching is a **bidirectional, case-insensitive substring match**: `rTitle.includes(dlTitle) || dlTitle.includes(rTitle)`.
|
||||||
|
5. For each match, resolves the series/movie, extracts user tags, checks ownership.
|
||||||
|
6. Returns only the requesting user's downloads (or all, if admin with `showAll=true`).
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
Start(["Download item"]) --> SQ{"Sonarr QUEUE\nmatch (title)"}
|
||||||
|
SQ -->|yes| SQR["Resolve series · extract user tag"]
|
||||||
|
SQ -->|no| RQ{"Radarr QUEUE\nmatch (title)"}
|
||||||
|
RQ -->|yes| RQR["Resolve movie · extract user tag"]
|
||||||
|
RQ -->|no| SH{"Sonarr HISTORY\nmatch (title)"}
|
||||||
|
SH -->|yes| SHR["Resolve series via seriesId"]
|
||||||
|
SH -->|no| RH{"Radarr HISTORY\nmatch (title)"}
|
||||||
|
RH -->|yes| RHR["Resolve movie via movieId"]
|
||||||
|
RH -->|no| Skip(["Skip — unmatched"])
|
||||||
|
|
||||||
|
SQR & RQR & SHR & RHR --> Tagged{"Tag matches\nrequesting user?"}
|
||||||
|
Tagged -->|yes| Include(["Include in response"])
|
||||||
|
Tagged -->|no| Skip
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Tag matching
|
||||||
|
|
||||||
|
Users are matched to downloads via Sonarr/Radarr tags:
|
||||||
|
|
||||||
|
1. **Exact match** — tag label (lowercased) === username (lowercased).
|
||||||
|
2. **Sanitised match** — handles Ombi tag mangling: `sanitizeTagLabel()` converts to lowercase, replaces non-alphanumeric chars with hyphens, collapses runs, trims.
|
||||||
|
|
||||||
|
#### Matched download object fields
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `type` | `'series'`/`'movie'`/`'torrent'` | Media type |
|
||||||
|
| `title` | string | Raw download title |
|
||||||
|
| `coverArt` | string/null | Poster URL from *arr |
|
||||||
|
| `status` | string | Download status |
|
||||||
|
| `progress` | string | Percentage complete |
|
||||||
|
| `size`/`mb`/`mbmissing` | string/number | Size info |
|
||||||
|
| `speed` | string | Current download speed |
|
||||||
|
| `eta` | string | Estimated time remaining |
|
||||||
|
| `seriesName`/`movieName` | string | Friendly media title |
|
||||||
|
| `episodes` | `{season, episode, title}[]` | Episodes covered (sorted); empty array if Sonarr has no data |
|
||||||
|
| `allTags` | string[] | All resolved tag labels on the series/movie |
|
||||||
|
| `matchedUserTag` | string/null | Tag label matching the requesting user |
|
||||||
|
| `tagBadges` | `{label, matchedUser}[]`/undefined | (Admin `showAll` only) each tag classified against Emby user list |
|
||||||
|
| `importIssues` | string[]/null | Import warning/error messages |
|
||||||
|
| `canBlocklist` | boolean | `true` if the current user may blocklist this download |
|
||||||
|
| `downloadPath` | string/null | (Admin) Download client path |
|
||||||
|
| `targetPath` | string/null | (Admin) *arr target path |
|
||||||
|
| `arrLink` | string/null | (Admin) Link to *arr web UI |
|
||||||
|
| `arrQueueId` | number/null | (Admin) Sonarr/Radarr queue record id |
|
||||||
|
| `arrType` | `'sonarr'`/`'radarr'`/null | (Admin) Which *arr service owns this queue entry |
|
||||||
|
| `arrInstanceUrl` | string/null | (Admin) Base URL of the *arr instance |
|
||||||
|
| `arrInstanceKey` | string/null | (Admin) API key for the *arr instance |
|
||||||
|
| `arrContentId` | number/null | (Admin) `episodeId` or `movieId` for triggering a new search |
|
||||||
|
| `arrContentType` | `'episode'`/`'movie'`/null | (Admin) Content type for search command |
|
||||||
|
| `addedOn` | number/null | (qBittorrent) Unix timestamp when torrent was added |
|
||||||
|
| `availableForUpgrade` | boolean/undefined | (History) `true` when outcome is `failed` but content is on disk |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Caching and Smart Polling
|
||||||
|
|
||||||
|
### 6.1 Cache Layer
|
||||||
|
|
||||||
|
`server/utils/cache.js` exports a singleton `MemoryCache` backed by a `Map`. Each entry carries an expiration timestamp. The cache is shared by the poller, webhook processor, and all route modules.
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
class MemoryCache {
|
||||||
|
get(key): any
|
||||||
|
set(key, value, ttlMs)
|
||||||
|
invalidate(key)
|
||||||
|
clear()
|
||||||
|
getStats(): CacheStats // per-key size, item count, TTL remaining
|
||||||
|
|
||||||
|
// Webhook metrics helpers
|
||||||
|
updateWebhookMetrics(instanceUrl)
|
||||||
|
getWebhookMetrics(instanceUrl): { eventsReceived, lastWebhookTimestamp, pollsSkipped }
|
||||||
|
getGlobalWebhookMetrics(): { lastGlobalWebhookTimestamp }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 Cache Keys
|
||||||
|
|
||||||
|
| Key | Content | TTL |
|
||||||
|
|-----|---------|-----|
|
||||||
|
| `poll:sab-queue` | `{ slots, status, speed, kbpersec }` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:sab-history` | `{ slots }` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:sonarr-queue` | `{ records }` with embedded `series` objects + `_instanceUrl`/`_instanceKey` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:sonarr-history` | `{ records }` — lightweight, no embedded objects | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:sonarr-tags` | `[{ instance, data: [{id, label}] }]` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:radarr-queue` | `{ records }` with embedded `movie` objects + `_instanceUrl`/`_instanceKey` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:radarr-history` | `{ records }` — lightweight | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:radarr-tags` | `[{ instance, data: [{id, label}] }]` | `POLL_INTERVAL × 3` |
|
||||||
|
| `poll:qbittorrent` | `[torrent, …]` | `POLL_INTERVAL × 3` |
|
||||||
|
| `history:sonarr` | `[record, …]` flat array with `_instanceUrl`/`_instanceName` | 5 min |
|
||||||
|
| `history:radarr` | `[record, …]` flat array with `_instanceUrl`/`_instanceName` | 5 min |
|
||||||
|
| `emby:users` | `Map<lowerName, displayName>` | 60 s |
|
||||||
|
|
||||||
|
When polling is disabled (`POLL_INTERVAL=0`), all `poll:*` TTLs fall back to **30 s** and data is fetched on-demand when the dashboard finds an empty cache entry.
|
||||||
|
|
||||||
|
### 6.3 Background Polling Modes
|
||||||
|
|
||||||
|
| Mode | `POLL_INTERVAL` | Behaviour |
|
||||||
|
|------|----------------|-----------|
|
||||||
|
| **Background** | `> 0` (e.g. `5000`) | Periodic fetch every N ms; SSE subscribers notified after each cycle |
|
||||||
|
| **On-demand** | `0` / `off` / `false` | Fetch triggered by first dashboard request when cache is empty; cached 30 s |
|
||||||
|
|
||||||
|
The poller uses a `polling` boolean flag to prevent concurrent cycles: if an interval fires while the previous poll is still running, the new invocation is skipped and logged.
|
||||||
|
|
||||||
|
### 6.4 Smart Polling Optimisation
|
||||||
|
|
||||||
|
When Sonarr/Radarr are configured to send webhooks to sofarr, the poller automatically reduces unnecessary API calls:
|
||||||
|
|
||||||
```
|
```
|
||||||
pollAllServices() called every POLL_INTERVAL ms:
|
pollAllServices() called every POLL_INTERVAL ms:
|
||||||
@@ -141,93 +593,335 @@ pollAllServices() called every POLL_INTERVAL ms:
|
|||||||
&& all instances have metrics.lastWebhookTimestamp within WEBHOOK_FALLBACK_TIMEOUT
|
&& all instances have metrics.lastWebhookTimestamp within WEBHOOK_FALLBACK_TIMEOUT
|
||||||
|
|
||||||
if shouldSkip:
|
if shouldSkip:
|
||||||
extend TTL of existing cached data ← no API calls made
|
extend TTL of existing cached data ← zero *arr API calls
|
||||||
increment metrics.pollsSkipped
|
increment metrics.pollsSkipped
|
||||||
log "[Poller] Skipping sonarr polling for N instance(s) with active webhooks"
|
log "[Poller] Skipping sonarr polling for N instance(s) with active webhooks"
|
||||||
else:
|
else:
|
||||||
fetch from *arr APIs → update cache
|
fetch from *arr APIs → update cache
|
||||||
```
|
```
|
||||||
|
|
||||||
**Result:** zero *arr API calls per poll cycle when webhooks are active and recent. Falls back automatically after `WEBHOOK_FALLBACK_TIMEOUT` minutes of silence (default: 10).
|
**Effect:** zero *arr API calls per poll cycle when webhooks are active and recent. The poller automatically falls back to full polling after `WEBHOOK_FALLBACK_TIMEOUT` minutes of silence (default: 10 minutes), ensuring the dashboard remains accurate even if webhooks stop arriving.
|
||||||
|
|
||||||
|
### 6.5 Active SSE Client Tracking
|
||||||
|
|
||||||
|
SSE connections are tracked precisely in `activeClients` (a `Map` keyed by `${username}:${connectedAt}`): registered on connect, removed on disconnect. The admin status panel shows each connected user and their connection duration. The `type: 'sse'` field distinguishes SSE clients from other connection types.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Cache Keys
|
## 7. Key Subsystems
|
||||||
|
|
||||||
| Key | Content | TTL |
|
### 7.1 Download Clients
|
||||||
|-----|---------|-----|
|
|
||||||
| `poll:sab-queue` | SABnzbd queue slots + status | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:sab-history` | SABnzbd history slots | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:sonarr-queue` | Sonarr queue records (with `_instanceUrl`) | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:sonarr-history` | Sonarr history records | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:sonarr-tags` | Sonarr tag list per instance | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:radarr-queue` | Radarr queue records (with `_instanceUrl`) | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:radarr-history` | Radarr history records | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:radarr-tags` | Radarr tag list | `POLL_INTERVAL × 3` |
|
|
||||||
| `poll:qbittorrent` | qBittorrent torrent list | `POLL_INTERVAL × 3` |
|
|
||||||
| `history:sonarr` | Sonarr history (on-demand, `/api/history/recent`) | 5 min |
|
|
||||||
| `history:radarr` | Radarr history (on-demand) | 5 min |
|
|
||||||
| `emby:users` | Emby user list | 60 s |
|
|
||||||
|
|
||||||
When polling is disabled (`POLL_INTERVAL=0`), all `poll:*` TTLs fall back to 30 s.
|
See [Section 3.1](#31-pluggable-download-client-architecture-pdca) for full detail. The client hierarchy is:
|
||||||
|
|
||||||
|
```
|
||||||
|
DownloadClient (abstract — server/clients/DownloadClient.js)
|
||||||
|
├── SABnzbdClient.js — Usenet; REST; API key auth
|
||||||
|
├── QBittorrentClient.js — Torrent; Sync API + fallback; cookie auth
|
||||||
|
├── TransmissionClient.js — Torrent; JSON-RPC; session-ID management
|
||||||
|
└── RTorrentClient.js — Torrent; XML-RPC; HTTP Basic Auth
|
||||||
|
```
|
||||||
|
|
||||||
|
`server/utils/qbittorrent.js` is a legacy compatibility shim that delegates to `QBittorrentClient`.
|
||||||
|
|
||||||
|
### 7.2 Queue & History Processing
|
||||||
|
|
||||||
|
**`server/utils/historyFetcher.js`** fetches history records from all Sonarr/Radarr instances for a configurable date window. Results are cached under `history:sonarr` / `history:radarr` for 5 minutes. Exports `classifySonarrEvent` / `classifyRadarrEvent` (returns `'imported'` | `'failed'` | `'other'`) and `invalidateHistoryCache`.
|
||||||
|
|
||||||
|
**`server/routes/history.js`** (`GET /api/history/recent`) returns recently completed (imported or failed) downloads filtered for the authenticated user. Supports `?days=N` (default `RECENT_COMPLETED_DAYS`, capped at 90) and `?showAll=true` for admins. Results are sorted newest first.
|
||||||
|
|
||||||
|
**`server/routes/dashboard.js`** (`POST /api/dashboard/blocklist-search`) removes a Sonarr/Radarr queue item with `blocklist=true` and immediately triggers an `EpisodeSearch` or `MoviesSearch` command. Non-admin users may only blocklist when import issues are present, or (for qBittorrent only) the torrent is over 1 hour old with less than 100% availability.
|
||||||
|
|
||||||
|
### 7.3 Dashboard & Frontend
|
||||||
|
|
||||||
|
The frontend is a **vanilla JavaScript SPA** with no build step. All logic resides in `public/app.js`, styled by `public/style.css`, and structured by `public/index.html`. Three CSS themes are available via the `data-theme` attribute on `<html>` and persist in `localStorage`:
|
||||||
|
|
||||||
|
- **Light** — Purple gradient header, white cards
|
||||||
|
- **Dark** — Dark surfaces, muted accents
|
||||||
|
- **Mono** — Monochrome, minimal colour
|
||||||
|
|
||||||
|
#### UI state machine
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
stateDiagram-v2
|
||||||
|
[*] --> SplashScreen : Page load
|
||||||
|
SplashScreen --> CheckAuth : checkAuthentication()
|
||||||
|
|
||||||
|
state CheckAuth <<choice>>
|
||||||
|
CheckAuth --> LoginForm : No session
|
||||||
|
CheckAuth --> Dashboard : Valid session
|
||||||
|
|
||||||
|
LoginForm --> Dashboard : Auth success (fade transition)
|
||||||
|
Dashboard --> LoginForm : Logout (stopSSE)
|
||||||
|
|
||||||
|
state Dashboard {
|
||||||
|
[*] --> Rendering
|
||||||
|
Rendering --> Rendering : SSE message → renderDownloads()
|
||||||
|
|
||||||
|
state SSEConnection {
|
||||||
|
[*] --> Connecting
|
||||||
|
Connecting --> Connected : First message
|
||||||
|
Connected --> Reconnecting : Connection lost
|
||||||
|
Reconnecting --> Connected : Auto-reconnect
|
||||||
|
Connected --> Connecting : showAll toggled
|
||||||
|
}
|
||||||
|
|
||||||
|
state StatusPanel {
|
||||||
|
[*] --> Closed
|
||||||
|
Closed --> Open : Click Status (admin)
|
||||||
|
Open --> Closed : Click close
|
||||||
|
Open --> Open : 5s timer refresh
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Key frontend functions
|
||||||
|
|
||||||
|
| Function | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| `checkAuthentication()` | On load: check session → show dashboard or login |
|
||||||
|
| `handleLogin()` | Authenticate, fade login → splash → dashboard |
|
||||||
|
| `startSSE()` | Open `EventSource` to `/stream`; handle incoming data |
|
||||||
|
| `stopSSE()` | Close `EventSource` and cancel reconnect timer |
|
||||||
|
| `renderDownloads()` | Diff-based card rendering (create/update/remove) |
|
||||||
|
| `createDownloadCard()` | Build DOM for a single card; renders tag badges, import-issue badge, blocklist button |
|
||||||
|
| `updateDownloadCard()` | Update existing card in-place (progress, speed, etc.) |
|
||||||
|
| `handleBlocklistSearch()` | Confirm dialog → POST `/blocklist-search` → update button state |
|
||||||
|
| `toggleStatusPanel()` | Show/hide admin status panel |
|
||||||
|
| `renderStatusPanel()` | Build status HTML (server, polling, SSE clients, cache) |
|
||||||
|
| `initThemeSwitcher()` | Light / Dark / Mono theme support |
|
||||||
|
| `loadHistory()` | Fetch `/api/history/recent`, store raw items, call `renderHistory()` |
|
||||||
|
| `renderHistory()` | Filter items by `ignoreAvailable` flag, render history cards |
|
||||||
|
|
||||||
|
#### Tag badge rendering
|
||||||
|
|
||||||
|
- **Regular user view** — a single accent-coloured badge showing the tag label that matched the current user's username (via `matchedUserTag`).
|
||||||
|
- **Admin `showAll` view** — all tags on the download are rendered using `tagBadges[]`: tags with no matching Emby user → amber badge (leftmost); tags matched to a known Emby user → accent badge showing the Emby display name (rightmost).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Security Model
|
## 8. Directory Structure
|
||||||
|
|
||||||
| Concern | Mechanism |
|
|
||||||
|---------|-----------|
|
|
||||||
| User authentication | Emby credentials → httpOnly HMAC-signed cookie |
|
|
||||||
| Session validation | `requireAuth` middleware on all `/api/dashboard`, `/api/history`, proxy routes |
|
|
||||||
| CSRF | Double-submit cookie (`X-CSRF-Token` header) on all state-changing routes |
|
|
||||||
| Webhook auth | Shared secret on `X-Sofarr-Webhook-Secret` header (webhook routes are outside CSRF) |
|
|
||||||
| Webhook input | `validatePayload()` allowlists event types; rejects invalid shapes |
|
|
||||||
| Webhook replay | 5-minute nonce cache keyed on `(eventType, instanceName, date)` |
|
|
||||||
| Rate limiting | 300 req/15 min (general), 10 fails/15 min (login), 60 req/1 min (webhook) |
|
|
||||||
| Secret leakage | `sanitizeError()` redacts all secrets from error messages and logs |
|
|
||||||
| Headers | Helmet v7: CSP nonce, HSTS, X-Frame-Options DENY, noSniff, Referrer-Policy |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Directory Structure (summary)
|
|
||||||
|
|
||||||
```
|
```
|
||||||
sofarr/
|
sofarr/
|
||||||
├── server/
|
├── server/
|
||||||
│ ├── app.js Express factory (imported by tests + index.js)
|
│ ├── app.js Express app factory — imported by tests and index.js
|
||||||
│ ├── index.js Entry point: logging, listen, start poller
|
│ ├── index.js Entry point: logging setup, server listen, poller start
|
||||||
│ ├── clients/ PDCA — one file per download client
|
│ ├── clients/ PDCA — one file per download client + retriever
|
||||||
|
│ │ ├── DownloadClient.js Abstract base class for all download clients
|
||||||
|
│ │ ├── QBittorrentClient.js
|
||||||
|
│ │ ├── SABnzbdClient.js
|
||||||
|
│ │ ├── TransmissionClient.js
|
||||||
|
│ │ ├── RTorrentClient.js
|
||||||
|
│ │ ├── PollingSonarrRetriever.js PALDRA — Sonarr retriever
|
||||||
|
│ │ └── PollingRadarrRetriever.js PALDRA — Radarr retriever
|
||||||
│ ├── routes/
|
│ ├── routes/
|
||||||
│ │ ├── auth.js Login / logout / csrf / me
|
│ │ ├── auth.js POST /login, GET /me, GET /csrf, POST /logout
|
||||||
│ │ ├── dashboard.js SSE stream, downloads, status, cover-art
|
│ │ ├── dashboard.js SSE /stream, /user-downloads, /user-summary, /status, /cover-art, /blocklist-search
|
||||||
│ │ ├── history.js Recently completed downloads
|
│ │ ├── history.js GET /api/history/recent
|
||||||
│ │ ├── webhook.js Webhook receiver (Phase 1–6)
|
│ │ ├── webhook.js POST /api/webhook/sonarr|radarr
|
||||||
│ │ ├── sonarr.js Sonarr API proxy + webhook management
|
│ │ ├── sonarr.js Sonarr API proxy + webhook management
|
||||||
│ │ └── radarr.js Radarr API proxy + webhook management
|
│ │ ├── radarr.js Radarr API proxy + webhook management
|
||||||
|
│ │ ├── emby.js Emby API proxy
|
||||||
|
│ │ └── sabnzbd.js SABnzbd API proxy
|
||||||
│ ├── middleware/
|
│ ├── middleware/
|
||||||
│ │ ├── requireAuth.js Cookie auth enforcement
|
│ │ ├── requireAuth.js httpOnly cookie auth enforcement
|
||||||
│ │ └── verifyCsrf.js Double-submit CSRF check
|
│ │ └── verifyCsrf.js Double-submit CSRF check (timing-safe)
|
||||||
│ └── utils/
|
│ └── utils/
|
||||||
│ ├── arrRetrievers.js PALDRA — Sonarr/Radarr fetch registry
|
│ ├── arrRetrievers.js PALDRA registry — Sonarr/Radarr fetch registry
|
||||||
│ ├── cache.js MemoryCache + webhook metrics helpers
|
│ ├── cache.js MemoryCache + webhook metrics helpers
|
||||||
│ ├── config.js Multi-instance config parser
|
│ ├── config.js Multi-instance config parser
|
||||||
│ ├── downloadClients.js PDCA registry + factory
|
│ ├── downloadClients.js PDCA registry + factory
|
||||||
│ ├── historyFetcher.js History fetch + event classification
|
│ ├── historyFetcher.js History fetch + event classification
|
||||||
|
│ ├── logger.js File logger (DATA_DIR/server.log)
|
||||||
│ ├── poller.js Smart background polling engine
|
│ ├── poller.js Smart background polling engine
|
||||||
│ ├── sanitizeError.js Secret redaction from errors
|
│ ├── qbittorrent.js Legacy compatibility shim → QBittorrentClient
|
||||||
│ └── tokenStore.js Emby token store (JSON file, atomic writes)
|
│ ├── sanitizeError.js Secret redaction from errors/logs
|
||||||
├── public/ Static SPA (HTML + CSS + vanilla JS)
|
│ └── tokenStore.js Emby token store (JSON file, atomic writes, 31-day TTL)
|
||||||
|
├── public/ Static SPA (served by Express)
|
||||||
|
│ ├── index.html HTML shell: splash, login, dashboard
|
||||||
|
│ ├── app.js All frontend logic
|
||||||
|
│ ├── style.css Themes, layout, responsive design
|
||||||
|
│ ├── favicon.ico / *.png Favicons
|
||||||
|
│ └── images/ Logo / splash screen assets
|
||||||
├── tests/
|
├── tests/
|
||||||
│ ├── setup.js Isolated DATA_DIR, SKIP_RATE_LIMIT
|
│ ├── README.md Testing approach and coverage targets
|
||||||
│ ├── unit/ Pure unit tests
|
│ ├── setup.js Global setup: isolated DATA_DIR, rate-limit bypass
|
||||||
|
│ ├── unit/ Pure unit tests (no HTTP)
|
||||||
│ └── integration/ Supertest + nock integration tests
|
│ └── integration/ Supertest + nock integration tests
|
||||||
├── docs/ARCHITECTURE.md Full deep-dive architecture documentation
|
├── .gitea/workflows/
|
||||||
├── ARCHITECTURE.md This file — concise reference
|
│ ├── ci.yml Security audit + test/coverage on every push/PR
|
||||||
├── SECURITY.md Threat model + hardening guide
|
│ ├── build-image.yml Docker image build and push
|
||||||
|
│ ├── create-release.yml Release tagging workflow
|
||||||
|
│ ├── docs-check.yml Markdown lint + Mermaid validation
|
||||||
|
│ └── licence-check.yml Production dependency licence check
|
||||||
|
├── Dockerfile Multi-stage production image (node:22-alpine)
|
||||||
|
├── docker-compose.yaml Example compose deployment
|
||||||
|
├── vitest.config.js Test runner configuration with per-file coverage thresholds
|
||||||
|
├── package.json Dependencies and scripts
|
||||||
|
├── ARCHITECTURE.md This document
|
||||||
|
├── SECURITY.md Threat model and hardening guide
|
||||||
├── CHANGELOG.md Version history
|
├── CHANGELOG.md Version history
|
||||||
└── .env.sample Annotated configuration template
|
└── .env.sample Annotated environment variable template
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*For complete API reference, data-flow diagrams, download matching pipeline, qBittorrent Sync API details, and deployment guidance see [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).*
|
## 9. Configuration and Environment Variables
|
||||||
|
|
||||||
|
### 9.1 Core Server
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|:--------:|---------|-------------|
|
||||||
|
| `PORT` | No | `3001` | Server listen port |
|
||||||
|
| `NODE_ENV` | No | — | Set to `production` for production logging and startup validation |
|
||||||
|
| `DATA_DIR` | No | `./data` | Directory for `tokens.json` and `server.log`. Must be writable. In Docker: `/app/data` (named volume). |
|
||||||
|
| `COOKIE_SECRET` | No* | — | Signs all session cookies with HMAC-SHA256. **Strongly recommended in production** (server exits on startup if unset in `NODE_ENV=production`). Generate with `openssl rand -hex 32`. |
|
||||||
|
| `TRUST_PROXY` | No | — | Express `trust proxy` setting. Set to `1` when behind a TLS-terminating reverse proxy (nginx, Caddy, Traefik) so `req.ip` and `req.secure` are correct. |
|
||||||
|
| `LOG_LEVEL` | No | `info` | `debug`, `info`, `warn`, `error`, `silent` |
|
||||||
|
| `RECENT_COMPLETED_DAYS` | No | `7` | Default lookback window for `/api/history/recent`. Overridable per-request via `?days=`. Capped at 90. |
|
||||||
|
|
||||||
|
### 9.2 TLS / HTTPS
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|:--------:|---------|-------------|
|
||||||
|
| `TLS_ENABLED` | No | `true` | Set to `false` to run plain HTTP (e.g. when TLS is terminated by a reverse proxy). |
|
||||||
|
| `TLS_CERT` | No | `certs/snakeoil.crt` | Path to TLS certificate (PEM). Defaults to the bundled self-signed snakeoil certificate. |
|
||||||
|
| `TLS_KEY` | No | `certs/snakeoil.key` | Path to TLS private key (PEM). |
|
||||||
|
|
||||||
|
### 9.3 Webhook
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|:--------:|---------|-------------|
|
||||||
|
| `SOFARR_WEBHOOK_SECRET` | Yes* | — | Shared secret validated on the `X-Sofarr-Webhook-Secret` header. Webhook endpoints reject all requests if this is not set. Generate with `openssl rand -hex 32`. |
|
||||||
|
| `SOFARR_BASE_URL` | Yes* | — | Public base URL of this sofarr instance (e.g. `https://sofarr.example.com`). Used by the one-click webhook configuration endpoints to tell Sonarr/Radarr where to send events. |
|
||||||
|
| `WEBHOOK_FALLBACK_TIMEOUT` | No | `10` | Minutes of silence after which the poller falls back to full polling even when webhooks were recently active. |
|
||||||
|
|
||||||
|
### 9.4 Polling
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|:--------:|---------|-------------|
|
||||||
|
| `POLL_INTERVAL` | No | `5000` | Background poll interval in ms. Set to `0`, `off`, or `false` to disable and use on-demand mode. |
|
||||||
|
|
||||||
|
### 9.5 Emby
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|:--------:|---------|-------------|
|
||||||
|
| `EMBY_URL` | Yes | — | Emby/Jellyfin base URL (e.g. `https://emby.example.com`) |
|
||||||
|
| `EMBY_API_KEY` | Yes | — | Emby API key — used by the poller to list users for tag badge classification |
|
||||||
|
|
||||||
|
### 9.6 Service Instances
|
||||||
|
|
||||||
|
All service instances support both a JSON array format (recommended) and a legacy single-instance format:
|
||||||
|
|
||||||
|
| Variable | Required | Format |
|
||||||
|
|----------|:--------:|--------|
|
||||||
|
| `SONARR_INSTANCES` | Yes* | JSON array |
|
||||||
|
| `SONARR_URL` + `SONARR_API_KEY` | Yes* | Legacy single-instance |
|
||||||
|
| `RADARR_INSTANCES` | Yes* | JSON array |
|
||||||
|
| `RADARR_URL` + `RADARR_API_KEY` | Yes* | Legacy single-instance |
|
||||||
|
| `SABNZBD_INSTANCES` | Yes* | JSON array |
|
||||||
|
| `SABNZBD_URL` + `SABNZBD_API_KEY` | Yes* | Legacy single-instance |
|
||||||
|
| `QBITTORRENT_INSTANCES` | No | JSON array (uses `username`/`password` not `apiKey`) |
|
||||||
|
| `RTORRENT_INSTANCES` | No | JSON array (URL must include the full XML-RPC path, e.g. `/RPC2`) |
|
||||||
|
|
||||||
|
\* Either `*_INSTANCES` or the legacy pair is required for each service.
|
||||||
|
|
||||||
|
#### JSON array instance format
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{ "name": "main", "url": "https://sonarr.example.com", "apiKey": "your-api-key" },
|
||||||
|
{ "name": "4k", "url": "https://sonarr4k.example.com", "apiKey": "your-4k-api-key" }
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
qBittorrent and rTorrent instances use `username` and `password` instead of `apiKey`.
|
||||||
|
|
||||||
|
Each instance receives an `id` derived from `name` (or index if unnamed), used as the key in PDCA and PALDRA registries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Security Model
|
||||||
|
|
||||||
|
### 10.1 Authentication and Sessions
|
||||||
|
|
||||||
|
| Concern | Mechanism |
|
||||||
|
|---------|-----------|
|
||||||
|
| **User authentication** | Emby credentials via `POST /Users/authenticatebyname`. A deterministic `DeviceId` (SHA-256 of username, first 16 chars) ensures Emby reuses the same session on every login. |
|
||||||
|
| **Session cookie** | `httpOnly`, `sameSite: strict`, `secure` when `TRUST_PROXY` is set. Payload: `{ id, name, isAdmin }` only — the Emby `AccessToken` is **never** sent to the browser. Signed with HMAC when `COOKIE_SECRET` is set. |
|
||||||
|
| **Token store** | Emby `AccessToken`s stored server-side in `DATA_DIR/tokens.json` (atomic writes, 31-day TTL, hourly pruning). Used only for server-side Emby logout. |
|
||||||
|
| **Session validation** | `requireAuth` middleware on all `/api/dashboard`, `/api/history`, and proxy routes. Returns `401` if the cookie is absent, tampered, or schema-invalid. |
|
||||||
|
| **CSRF protection** | Double-submit cookie pattern. `verifyCsrf` middleware compares the `csrf_token` cookie against the `X-CSRF-Token` request header using `crypto.timingSafeEqual`. Applied to all state-changing requests (`POST`, `PUT`, `PATCH`, `DELETE`) under `/api/*` except auth and webhook routes. |
|
||||||
|
| **Remember-me** | `rememberMe: true` → persistent cookie, `Max-Age` 30 days. `rememberMe: false` → session cookie (expires on browser close). |
|
||||||
|
|
||||||
|
### 10.2 Webhook Security
|
||||||
|
|
||||||
|
| Concern | Mechanism |
|
||||||
|
|---------|-----------|
|
||||||
|
| **Secret validation** | Every webhook request must carry `X-Sofarr-Webhook-Secret` matching `SOFARR_WEBHOOK_SECRET`. Absent or wrong secret → `401`. Webhook endpoints function outside the CSRF middleware (they are not browser-initiated). |
|
||||||
|
| **Rate limiting** | Dedicated `webhookLimiter`: 60 req/min per IP (stricter than the general 300 req/15 min limiter). |
|
||||||
|
| **Payload validation** | `validatePayload()` enforces: JSON object body, `eventType` as a non-empty string ≤ 64 chars, `eventType` in the allowlist, `instanceName` as string if present. Rejects with `400` on any violation. |
|
||||||
|
| **Replay protection** | `isReplay()` caches a composite key `{eventType}:{instanceName}:{date}` for 5 minutes. Duplicate events within that window are acknowledged with `200 { received: true, duplicate: true }` and not processed. |
|
||||||
|
|
||||||
|
### 10.3 Additional Security Measures
|
||||||
|
|
||||||
|
| Concern | Mechanism |
|
||||||
|
|---------|-----------|
|
||||||
|
| **Rate limiting** | 300 req/15 min general (all API routes); 10 failed attempts/15 min login limiter; 60 req/1 min webhook limiter. |
|
||||||
|
| **Secret leakage** | `sanitizeError()` (`server/utils/sanitizeError.js`) redacts secrets from error messages and logs: URL query-param secrets (`apikey=`, `token=`), HTTP auth headers (`Authorization:`, `X-Emby-Authorization:`), Bearer tokens, and basic-auth credentials in URLs. |
|
||||||
|
| **HTTP headers** | Helmet v7: CSP with per-request nonce (`crypto.randomBytes(16)` for inline styles/scripts), HSTS, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy`, `Permissions-Policy`. |
|
||||||
|
| **Body size** | `express.json` body limit: 64 KB. |
|
||||||
|
| **Authorisation matrix** | Regular users see only their own downloads. Admins can view all users, see paths and *arr links, and blocklist any download. Non-admins can only blocklist when import issues exist or (for qBittorrent) the torrent is >1 h old with <100% availability. |
|
||||||
|
| **Container security** | Docker image runs as the non-root `node` user (UID 1000). `/app/data` is owned by `node`. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Technology Stack
|
||||||
|
|
||||||
|
### Runtime and Framework
|
||||||
|
|
||||||
|
| Layer | Technology | Notes |
|
||||||
|
|-------|-----------|-------|
|
||||||
|
| Runtime | Node.js 22 (Alpine) | LTS; ESM-ready; V8 coverage built-in |
|
||||||
|
| Framework | Express 4.x | HTTP server, routing, middleware |
|
||||||
|
| HTTP client | axios 1.x | External API communication |
|
||||||
|
| XML-RPC client | xmlrpc 1.3.2 | rTorrent communication |
|
||||||
|
| Frontend | Vanilla JS + CSS | SPA, no build step required |
|
||||||
|
| Containerisation | Docker multi-stage (node:22-alpine) | Non-root `node` user; minimal image |
|
||||||
|
| Logging | Custom logger + `console.*` redirection | File + stdout with configurable levels |
|
||||||
|
|
||||||
|
### Security Middleware
|
||||||
|
|
||||||
|
| Package | Version | Purpose |
|
||||||
|
|---------|---------|---------|
|
||||||
|
| `helmet` | 7.x | HTTP security headers (CSP nonce, HSTS, referrer policy, frame options) |
|
||||||
|
| `express-rate-limit` | 7.x | General, login, and webhook rate limiters |
|
||||||
|
| `cookie-parser` | 1.x | Signed cookie support (HMAC via `COOKIE_SECRET`) |
|
||||||
|
|
||||||
|
### Auth and Session
|
||||||
|
|
||||||
|
| Component | Technology | Details |
|
||||||
|
|-----------|-----------|---------|
|
||||||
|
| Identity provider | Emby / Jellyfin API | `POST /Users/authenticatebyname` |
|
||||||
|
| Session cookie | `httpOnly` + `sameSite: strict` | Signed when `COOKIE_SECRET` is set |
|
||||||
|
| CSRF protection | Double-submit cookie | `csrf_token` cookie + `X-CSRF-Token` header; `crypto.timingSafeEqual` |
|
||||||
|
| Token store | JSON file (`DATA_DIR/tokens.json`) | Atomic writes, 31-day TTL, hourly pruning |
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
| Tool | Version | Purpose |
|
||||||
|
|------|---------|---------|
|
||||||
|
| `vitest` | 4.x | Test runner with V8 coverage; per-file coverage thresholds in `vitest.config.js` |
|
||||||
|
| `supertest` | 7.x | HTTP integration testing against the Express app factory |
|
||||||
|
| `nock` | 14.x | HTTP interception at Node layer (compatible with CJS `require('axios')`) |
|
||||||
|
|
||||||
|
### CI/CD
|
||||||
|
|
||||||
|
| Workflow file | Trigger | Purpose |
|
||||||
|
|---------------|---------|---------|
|
||||||
|
| `ci.yml` | Every push / PR | `npm audit --audit-level=high` + full test suite with V8 coverage |
|
||||||
|
| `build-image.yml` | Push to `release/**` or `develop` | Build and push Docker image to registry |
|
||||||
|
| `create-release.yml` | Tag push (`v*`) | Generate release notes and create a Gitea release |
|
||||||
|
| `docs-check.yml` | Push / PR touching `**.md` | Markdown lint + Mermaid diagram parse validation |
|
||||||
|
| `licence-check.yml` | Push / PR touching `package.json` | Verify production dependency licences are MIT-compatible |
|
||||||
|
|||||||
1716
docs/ARCHITECTURE.md
1716
docs/ARCHITECTURE.md
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user