I see no point in making wooden walls breakable.
I haven't played Rust, which means I don't know how everything works. But if people can break into your house it kind of defeats the hardcore objective of raiding somebody. Without trying or stressing it makes the game more easy. Which I personally think would ruin everything.
if I was in a post-apocolyptic zombie situation, I would use any means to enter your house, kill you, and take everything you have. Even if that means busting through the wall with an axe.