Okta’s Red Team discovered HollowByte, a Denial of Service vulnerability in OpenSSL allowing any remote unauthenticated attacker to force servers to allocate disproportionate memory by sending a malicious 11-byte TLS payload. The TLS handshake’s ClientHello leads OpenSSL to allocate up to 131 KB based on an unvalidated 3-byte length declaration in the 4-byte header before data arrives, blocking the worker thread indefinitely.
This compounds with glibc’s memory handling, which retains freed small-to-medium allocations, fragmenting the heap and permanently bloating the server’s Resident Set Size even after disconnection. Tests showed unpatched NGINX servers being OOM-killed or locking up significant RAM percentages, bypassing connection-limiting defenses. The only recovery is process termination.
OpenSSL fixed it via incremental buffer growth in v4.0.1 and backports to older versions like 3.6.3, released silently over five weeks prior on June 9th as a hardening rather than CVE. Dependent software like Apache, NGINX, Node.js, and databases must update, but many appliances like Synology NAS remain on outdated versions, risking easy DoS until rebooted and patched. This highlights challenges for manually updated network gear amid rising AI-driven vulnerability discoveries.