He accused parents with more than two offspring of being "irresponsible" and criticised green groups for shying away from the issue in debate.
But family campaigners likened his remarks to China's one-child policy and suggested they could encourage a further rise in abortions.
His comments were also dismissed as "absolutely barmy" by the Conservative MP Ann Widdecombe, who said that Britain's problem was not too many children but too few.
The former Green Party politician, who is now chairman of the Government's Sustainable Development Commission, accused other environmentalists of "betraying" their members by not openly voicing calls for people to limit the number of children.
"I am unapologetic about asking people to connect up their own responsibility for their total environmental footprint and how they decide to procreate and how many children they think are appropriate," he said.
"I think we will work our way ...