By Ollie Stone-Lee BBC News Online political staff in Blackpool

The ex-Asda boss said the attendance at the party's annual conference in Blackpool was the lowest for "many, many decades".

Speaking at a Tory Reform Group fringe meeting at the conference, he said the membership had to be the driving energy behind the policy, but estimated it had dwindled to about 250,000.

Mr Norman, who played a key role under William Hague's leadership, said some of his parliamentary colleagues no longer "have their hearts" in the job.

Tory MPs should face mandatory reselection contests before party members, he said, pointing out that only MPs and judges "go on and on and on until they drop".

Mr Norman praised the "fantastic job" being done being done by Iain Duncan Smith - his concerns were instead about the party's internal organisation.

Smaller leadership

He also called for a shake-up of the party's leadership arrangements. The shadow cabinet could organise activity inside Parliament, he said, but added that was an "absurd" way of organising a national party.

"You cannot possibly run the politics of the party among 23 people," he said.

Instead, a much smaller executive should take overall charge.

That would mean having three or four top spokesmen to deliver the Tory message through the media, rather than using a range of shadow ministers for the different briefs tackled in Parliament.

There's no such thing as 'not our territory' - it's all our territory Archie Norman

The Tunbridge Wells MP was at the forefront of the reforms of the party organisation introduced under Mr Hague, but he admitted the transformation they had tried to achieve had failed to materialise.

"The reality is that reform programme was still born," he said. "If I'm honest about it, I think we landscaped the garden but the grass never grew."

Mr Norman argued the party could not afford to be complacent, saying the situation was "dire".

"The party is in a pretty parlous state at the moment."

Membership worries

Official figures might put Tory membership at about 300,000, but the real figure was probably 50,000 lower and the average age was about 65.

Such organisational matters were important, he argued, pointing to the way many areas of Britain now did not have any Conservative presence at all.

"The thought-building that drives a political party is affected by who we represent - that's why it's so serious," said Mr Norman.

He called for minimum membership subscriptions to be dropped and for the party to recruit people who shared Tory values, rather than for social activities.

Mr Norman also criticised the party's campaign in the Brent East by-election - where the Tories were pushed into third place in what he branded a "fiasco".

He and other MPs were sent messages asking them to campaign in the constituency a few days before the polling day but "that was about three months late".