Why raycast is ignorring collision with certain parts of an object?

I have a simple script attached to the player that makes it stay at an equal distance from the ground always.

void Update() {     RaycastHit hit;     if (Physics.Raycast(transform.position, Vector3.down, out hit))     {         transform.position = new Vector3(hit.point.x, hit.point.y + offset, hit.point.z);     } } 

But for some reason it ignores collision as you can see here: https://youtu.be/8X0pw5tkjpE

How can I get rid of this problem?