Speaking at a fringe event at Labour Party conference, organised by housing association, Moat, Benn said the policy simply "penalises" people.
Part of the shake-up to the welfare system, the 'bedroom tax' will cut the amount of benefit that people can get if they are deemed to have a spare bedroom in their council or housing association home.
He said the government had to recognise that these are "people's family homes", their children have grown up in the house and what appears to be an "empty room" is often used by extended family.
Brian Johnson, chief executive of Moat reiterated Benn's concern saying the tax will "affect 15 per cent of tenants".
"Structurally we haven't got the right homes for those people to move into.
"I don't think one bedroom homes are a good housing solution for anyone", he said.
Referring to the introduction of direct payment of benefits, Johnson said he "understands the philosophy behind it", but that it will result in a "much higher level of arrears for housing associations".
"Direct payment of benefits will create a real lending problem for us and affect our ability to create new homes in the future", he said.
Former housing and planning minister, John Healey MP, said to "its cost" the government is finding that welfare cannot be reformed without reforming housing.
"You cannot deal with the problems and mismatch between demand and supply in housing without dealing with the distortionary effects within housing benefit", he said.
Benn reiterated this saying the only way the problem can truly be addressed is by "getting the housing supply up".
Leslie Murphy OBE, chief executive of Crisis, said it was crucial to have a welfare system that "supports the vulnerable to live in decent homes".
She also called for reform of the private rented sector so that it is able to adapt to the "enormous pressure it is under".
"The private rented sector is growing enormously. It now makes up 16.5 per cent of the housing market and is likely to overtake the social sector", she said.
However Murphy was keen to stipulate the protections that are often not afforded to vulnerable young people within the housing market.
"You actually frankly have more rights if you go out and buy a toaster than if you are a private tenant", she said.