It's exactly that: the trickiness around debugging is the main thing that feels like it's got some barriers compared to a turnkey solution in an IDE. I heard VS Code and Godot was available until I realized that the LSP and debugger for Godot 4.x was unusable for months until the recent refactor.
Don't get me wrong though, I am totally using VS Code for my Rust projects. It just isn't a turnkey solution that I'd recommend to someone if they just want to hit "New project" and do the whole write-compile-debug loop without needing to understand anything. (I had also used it a while back prior to rust-analyzer being the main go-to extension, I think...)