Hi Leth,
I had a few issues like that. After playing around with the settings I managed to smooth things out a lot. Here's my setup now:
As you can see, I now use the Vertical input instead of a dedicated Climb button. It feels smoother and more natural to climb up things just by walking...