Post

Is my NodeJS container supposed to be running netcat? 😅 I usually wake up sipping coffee, not choking on it seeing my cloud usage, but I made the most of the situation by doing some amateur forensics. #selfhosting has its ups and downs...

Had to sift through some metrics, but eventually found the outlier, a NextJS container using 2 vCPU and 3GB ram. Once I got to a shell, the problem was pretty clear: in addition to the expected Node processes, there was a process running from /tmp/.libsystem/node and—uh oh! Netcat.

Netcat (nc) is a utility for making network connections, but is also used—as in this case—to establish remote access by malicious actors. It's convenient for that, too, because it's already present on most Linux systems.

I tried to inspect its available /proc/<pid> info, but either got nothing back or permission denied. Not sure what was up with that, but I'm sure I have more to learn there. Netstat showed a connection to we.love.servers.at.ioflood​.net—super legit looking. I suspect C2, but I left it alone.

As more of the services we rely on do awful things, some of us are inclined to selfhost out of spite, but as this incident shows, there are risks. This was isolated, but could have been much worse. It's easy to miss an image update that patches a CVE, or not have time to set up alerting.

In this case, it looks like there was probably a remote code execution vulnerability in NextJS that allowed an attacker to steal my hosting resources to mine crypto. I have major respect for folks doing this professionally, using the knowledge to layer security and detect attacks as they occur.