

It’s just another way of minimizing your attack surface. It’s pretty much the same as hiding behind a barrier when being shot at, you stick yourself out as little as possible.
In the same way it also helps to change your SSH port to somewhere in the high numbers like 38265. This is anecdotal of course, but the amount of attacks on SSH went down by literally 99% by just changing the port like that
Then you accept only keys, you lock down root (so the username must be guessed as well) and yeah, you’re safe.
They could observe a connection to the server, big difference. If the site is on a WordPress domain, that IP might lead to a load balancer that manages hundreds of sites.
Of course the reverse is also true, so for for example Facebook, of you hit one of their IPs, then its obvious what you’re accessing