Removing shoulder pads or size down?

Bought a vintage wool blend blazer a few months ago and it was an almost perfect fit. Since then I've lost enough weight that the change in fit is noticeable, especially at the shoulder (it's likely a 40R, I'm 5'9" and 39" at the chest) The shoulder pads can be moved around but are otherwise seen in, is it worth just getting them removed, or should I try to just get the whole thing taken in?