That's exactly what I said some months ago, when capone was first seen and we still didn't know where he was. If he has the power to shrink attacks aimed at him he can be really powerfull.
Rememer how in the first panel we saw his power, the cannon bullets and soldiers got to their normal size when passing through a barrier?
If that barrier works the same way when something goes towards him, imagine how every punch becomes useless (or he could even open a door in his body and let the punch just pass through his body)
Yeah, that can be seen accurately in the manga as well
And i bet post time skip his range increased too, just like Law's rooms became much bigger. It's possible that simply trying to engage him in an even remotely close range fight will get you affected by his ability.
On top of that, i think he might be getting other advantages while fighting opponents that he manages to trap inside himself. Maybe that tiny avatar he can create there is incredibly strong, or he can use reality warping powers. Maybe the reality powers are enabled by the Awakening of his DF, which would be a possible interpretation of him being the absolute "King" of the Castle, having godlike powers within.
Physics-wise, the attacks coming at him may not be weakened and even get much stronger if the change in size doesn't mean the change in mass. In short, their speed and mass stay the same and now they are even deadlier cause the area of contact is much smaller => more pressure.
But yeah, all of those possibilities make his fruit extremely potential. Making incoming attacks weaker by decreasing their size or and even let them pass through him (if his opponent is Luffy who can stretch) by opening two...windows on his castle then trap the arm in there. That must be annoying as fuck. Although that depends on how far his range is and how quick the shrinking rate is. But there you have some effective defensive barrier.
The absolute power within his castle is something I also thought about. Kinda like how Shunsui's shikai makes everyone obey his game rule within a range.
But the thing is...he actually doesn't fight alone. His crew is inside. With preparation, the dude is very hax cause if he armed himself a bunch of Pacifista inside and sea stone bullets Man, he can shoot from anywhere on his body. You catch his hands and think ''he can't shoot me now'' then suddenly you see a door opened on his stomach and a bunch of sea stone bullets or Pacifista laser shot at you. Bye Bye DF users!