Not a terrible assumption, people who are willing to stalk you that hard are potentially violent or unhinged enough to try violence as a last resort even if they suck at fighting.
I think Vesper's assumption is reasonable, Altare did martial arts and can do some pretty fast push ups as was shown in one of his minecraft streams, Magni goes to the gym often, Axel used to be a really active guy in his youth and is probably still at least in decent shape, Vesper bikes actively as a hobby and rode for thousands of miles on his bike during his wandering hobo saga.
So they could probably handle some fat neet idiot, unless they brought a gun or something, as none of them seem to be the stereotypical 200+ pound Dorito finger weeb gamer neckbeard. These aren't horribly out of shape men from what they've said about their activity habits, relative to their audience they're probably very in shape.