Holly crap, you're right, I did have two identical capsule colliders parented to my character... No idea how that happened but after removing one of them, the performance improved substantially.
Thanks, Justin! And sorry for wasting your time because of my stupidity ?