commitments into account, the Court of Appeal said.
Heathrow said it would challenge the decision, but the government said it would not appeal.
The judges said that in future, a third runway could go ahead, as long as it fits with the UK's climate policy.
The case was brought by environmental groups, councils and the Mayor of London.
There were "whoops and jumps of jubilation from environmentalists outside the court room" after the judgement, BBC environment analyst Roger Harrabin reported.
Transport Secretary Grant Shapps tweeted that the government would not appeal against the ruling.
He said in an interview that it was "for Heathrow and the courts to decide" whether the expansion should go ahead.
"This government is absolutely committed [from] the Prime Minister down to airport expansion, but, we want to make sure that expansion is environmentally friendly," he said.
Heathrow said it would challenge the court's decision at the Supreme Court, saying it was "confident that we will be successful".
"In the meantime, we are ready to work with the government to fix the issue that the court has raised," a Heathrow spokesperson said.
"Expanding Heathrow, Britain's biggest port and only hub, is essential to achieving the prime minister's vision of global Britain. We will get it done the right way, without jeopardising the planet's future."
Friends of the Earth, one of the environmental groups that brought the case, said the ruling was "an absolutely groundbreaking result for climate justice".
Will Rundle, head of legal at the campaign group, said: "This judgment has exciting wider implications for keeping climate change at the heart of all planning decisions.
"It's time for developers and public authorities to be held to account when it comes to the climate impact of their damaging developments."
Greenpeace said the government needed to "permanently ground Heathrow's expansion plans".
Greenpeace UK's executive director, John Sauven, said: "The third runway is already on its knees over costs, noise, air pollution, habitat loss and lack of access, and now Heathrow has yet another impossibly high hurdle to clear.
"Boris Johnson should now put Heathrow out of its misery and cancel the third runway once and for all. No ifs, no buts, no lies, no U-turns."
MPs voted overwhelmingly to support Heathrow expansion in 2018. Before Boris Johnson became prime minister, he pledged in 2015 to lie down "in front of those bulldozers and stop the building, stop the construction of that third runway" at Heathrow.
This judgement will fly round the globe quicker than a 747.
More and more politicians worldwide are pledging to make their economies climate neutral, but the Appeal Court judges have shown exactly what this intention implies.
From now on, every infrastructure spending decision in the UK could face legal challenge if it doesn't comply with the Climate Change Act, which mandates virtually zero emissions by 2050.
It's not clear that's what MPs intended when they signed up to the 2050 target, but in today's court ruling, it's what they've got.
Should the prime minister decide not to appeal against the ruling, it may also call into question aviation expansion in other airports, as there is currently no viable technical fix to make planes zero-carbon and carbon offsetting schemes proposed by the aviation industry are contentious.
Environmentalists see today as the day the reality of climate change finally dawned.
The Court of Appeal found that the government had not followed UK policy when backing the controversial expansion plans.
It said that the government had a duty to take into account the Paris climate agreement, which seeks to limit global warming.
It was "legally fatal" to the government's Heathrow expansion policy that it did not take those climate commitments into account, the judges said.
The British Chambers of Commerce (BCC) said business communities in the UK would be "bitterly disappointed that plans for a world-leading hub airport are now at risk".
"Without expansion, firms risk losing crucial regional connectivity and access to key markets across the world," said BCC director general Adam Marshall.
Industry lobby group the CBI said that while "all major projects must be consistent" with net zero carbon emissions by 2050, "it's clear that the government and aviation industry need to work closely to agree a robust decarbonisation plan".
"It is vital that the government and Heathrow work closely together to remedy the fair concerns raised by the judgement and keep this project on track. Opportunities for future trade will not wait," said Josh Hardie, CBI deputy-director general.
But airline group IAG, which owns British Airways, said: "We have always said the environmental impact and cost of Heathrow expansion needs independent review. The airport cannot be trusted. Its original £14bn cost for expansion is now £32bn."