Does it work with other height maps? It's possible that the tool you used for drawing that was still leaving some black spots, which would be the same height as your ground, assuming your ground height is 0, making it seem like there's a hole. If you move your camera there though, you'd probably see it go down to the ground, then go back up to the next position.
