(1) Should everyone, regardless of income, have access to housing in every neighborhood?
(2) What is the minimum acceptable quality of a house?
These are questions whose answers come with tradeoffs. With respect to (1): suppose we decide that everyone should, if they wish, be able to live in a house on a beach in Laguna (as it happens, the Southern California beach town I most enjoy visiting). Such a policy would be costly, and has implications for the distribution of other goods. But there are reasons to think it is socially desirable for people of mixed income to live together. The correct answer will involve normative judgments we make as a society, but we need to make these judgments explicitly. I don't know that we do that. Personally, I think everyone should be able to live in a safe neighborhood, be in a place where they can send their kids to a decent, publicly funded, school, and have a reasonable commute to work (30 minutes or less? 45?) within their choice set. If people choose to live further away, that is their business.
As for (2), in the US context, I would think the minimum acceptable house would have indoor plumbing, clean water (I wish I could say this was a given), good sanitation, reliable electricity, a minimum amount of floor space per person (although I am not sure what that is), and be very, very fire resistant. I am perhaps leaving something out, but anything beyond an agreed upon minimum adds to the cost of providing housing. Again, I don't think when we discuss housing we discuss the tradeoffs enough.