Memory exhaustion during a peak slot.
During a peak slot, new file uploads stopped progressing. The Node process became unresponsive to new HTTP traffic, the LibreOffice conversion queue stalled, and a small number of candidates near the end of their window were unable to finalize submissions before the server stopped accepting writes.
A subset of candidates — many encountering a structured exam environment for the first time — produced oversized documents, often Word and PowerPoint files containing full-resolution embedded images and screenshots. Each conversion spawned a headless soffice process. When several of these oversized conversions ran concurrently, their combined working set exhausted system RAM, pushing the box into heavy swap and starving the Node event loop.
- Process recycle on the conversion worker to release leaked memory.
- Serialized the conversion queue — one heavy convert in flight per node.
- Deferred oversized files to off-peak conversion so the live slot stayed responsive.
- Coordinated with invigilators to extend the affected candidates’ submission window by a small buffer.
- Pre-upload validation for file size and embedded media before bytes hit disk.
- Conversion concurrency cap tuned to available RAM, not core count.
- Backpressure in the upload API so the browser sees queueing instead of timeouts.
- Per-file conversion telemetry, surfaced before each subsequent slot.
No submission was lost. Every candidate who reached the upload step has a row in Submission, a signed PDF on disk, and a final merged artefact ready for evaluation. The atomic flag flip in the submission transaction made partial state impossible — either the submission was complete or it wasn’t, never half-counted.