StageFlow

Accepting untrusted inputs safely from both URLs and uploaded ZIP archives

I validate every resolved IP against a three-tier classification covering 17 CIDR ranges (cloud metadata at 169.254.169.254, RFC1918, link-local, multicast, CGNAT) before any NATS message is published. A single blocked IP in a DNS response rejects the entire URL. For ZIPs, the extractor enforces five checks before extracting a byte: 5,000 max entries, 100x max expansion ratio, 1 GiB total uncompressed size, 250 MiB per-entry limit, and path traversal plus NUL-byte rejection.

Making NATS at-least-once delivery safe for state mutations

RecordScannerCompletion opens a PostgreSQL transaction, reads completed_scanners, checks if the scanner is already present, and only writes if it is not. Two concurrent redeliveries race on the same row, and the loser finds the scanner already recorded. CreateJobIfAbsent uses INSERT ON CONFLICT DO NOTHING with a boolean return. Idempotency lives in the database transaction, not in a separate dedup layer.

Coordinating scanners that start and finish independently

I persist the expected scanner set in PostgreSQL, run one container per scanner inside a job pod, and only advance the state machine once every scanner has reported success or failure. The allComplete boolean is computed inside the same transaction that records each completion, so it stays accurate regardless of delivery order or count.

Detecting scanner failures when containers exit without publishing NATS events

Each StartScanner call ends with spawnMonitorContainer, which launches a goroutine that blocks on WaitContainer via the Podman socket. If the container exits with a non-zero code, the goroutine fetches the last 500 bytes of logs and calls failJobSafe. A separate deadline sweeper runs every 30 seconds, catching jobs stuck in EXTRACTING for over 5 minutes or SCANNING for over 30 minutes.

Keeping real-time status useful when browser connections are unreliable

I use an SSE hub with buffered channels and a three-select non-blocking write that evicts one pending event rather than blocking the publisher on a slow consumer. A 15-second keepalive heartbeat prevents proxy idle timeouts. On each new connection the handler sends a full status snapshot before entering the event loop, so reconnecting clients recover immediately without tracking missed events.

Producing a useful report when some scanners fail but others succeed

DecideScanFailureCompletion checks whether at least one ScannerResult has Success: true. If yes, it returns ScanFailureCompleteWithPartialResults and aggregation proceeds with whatever succeeded. Failed scanners are recorded in the report with status: failed and their error message. Summary counts come from successful results only, so the totals stay internally consistent.

Merging heterogeneous scanner outputs into one actionable report without hiding real findings

I normalize results into a unified contract and deduplicate with a conservative static equivalence table of about 15 known cross-scanner overlaps. Merging only happens when issues share both a canonical rule ID and the same pageId. Issues on different pages pass through unchanged. The higher-priority scanner's version is kept and annotated with alsoDetectedBy.