£150 fines for drivers caught using their mobile phones
Patrick McLoughlin said he wanted to make driving while using a mobile device "a social taboo like not wearing a seatbelt".
Figures show drivers using mobiles were responsible for just shy of 500 road accidents last year, the highest number on record.
“We will take action to tackle this persistent problem, with an emphasis on the most serious offenders. The message is clear: keep your hands on the wheel, not your phone. If you keep taking calls while at the wheel, you could end up being banned from the road," Mr McLoughlin said.
The proposals will be put out to a public consultation next year before the changes are made.
Under the plans, first-time offenders will be made to attend an educational course, but those who repeatedly flout the law face the enhanced fine, along with four penalty points, rather than three.
Drivers of heavy goods vehicles face a tougher penalty regime, Transport Minister Andrew Jones said, because the potential consequences of a crash are much more serious.