Boris Johnson spoke to the German chancellor earlier about the proposals he put forward to the EU - but the source said she made clear a deal based on them was "overwhelmingly unlikely".
They also claimed she said a deal would never be possible unless Northern Ireland stayed in a customs union.
Mrs Merkel's spokesman said they did not reveal confidential conversations.
The President of the European Council, Donald Tusk, sent a public tweet to Mr Johnson, writing: "What's at stake is not winning some stupid blame game. At stake is the future of Europe and the UK as well as the security and interests of our people."
Ireland's Tánaiste (Deputy Prime Minister), Simon Coveney, tweeted it was "hard to disagree" with Mr Tusk, adding: "We remain open to finalise a fair Brexit deal, but need a UK government willing to work with EU to get it done."
Scotland's First Minister and leader of the SNP, Nicola Sturgeon, said Downing Street's response to the phone call was an "attempt to shift the blame for the Brexit fiasco", while Labour called it a "cynical attempt to sabotage the negotiations".
Shadow Brexit secretary Sir Keir Starmer said Mr Johnson "will never take responsibility for his own failure to put forward a credible deal", and called on Parliament to "unite prevent this reckless government crashing us out of the EU".
Officially, the prime minister's spokesman said talks with the EU were "at a crucial point", but denied they were over.
Latest on Brexit deal talks
Is no-deal about to become PM's policy?
PM calls on EU to 'thrash out' Brexit objections
Judge dismisses no-deal Brexit court move
What is in Boris Johnson's Brexit plan?
The PM has insisted the UK will leave the EU on the Brexit deadline of 31 October, with or without a deal.
That is despite legislation passed by MPs last month, known as the Benn Act, which requires Mr Johnson to write to the EU requesting a further delay if no deal is signed off by Parliament by 19 October - unless MPs agree to a no-deal Brexit.
Mr Johnson sent new proposals for a deal to Brussels last week, with the key focus being on replacing the so-called backstop - the policy negotiated by Theresa May and the EU to prevent a hard border returning to the island of Ireland - which has long been a sticking point.
After presenting the new proposals, government sources hoped the UK might be able to enter an intense 10-day period of negotiations almost immediately, with the aim of coming to a final agreement at an EU summit on 17 October.
The EU pledged to examine them carefully, but a number of senior figures, including Irish Taoiseach Leo Varadkar, warned they did not form the basis for deeper negotiations - even if they believed a deal could still be done.
French President Emanuel Macron said the EU would decide at the end of the week whether a new deal was possible.
But after the phone call on Tuesday morning, the No 10 source said it had been a "clarifying moment", adding: "Talks in Brussels are close to breaking down, despite the fact that the UK has moved a long way."
No-one really wants to comment directly on this phone call - certainly not Berlin. But talking to EU officials and diplomats in Brussels, there is considerable scepticism.
That's because the words attributed to Angela Merkel do not reflect the EU's agreed language.
For one, Mrs Merkel and the EU have repeatedly said they will keep talking to the last second and will not pull the plug before that.
And secondly, the No 10 source claims the EU wants to keep Northern Ireland permanently "trapped" in the customs union - Brussels insists it doesn't want that at all, it just wants the option for Northern Ireland stay inside temporarily until something else is worked out.
So as I say, scepticism. It could be a misinterpretation or it could be a deliberate bit of spin, because we're now entering into a blame game about whose fault it is that progress isn't being made.
Under Mr Johnson's proposals, which he calls a "broad landing zone" for a new deal with the EU:
Northern Ireland would leave the EU's customs union alongside the rest of the UK, at the start of 2021
But Northern Ireland would continue to apply EU legislation relating to agricultural and other products, if the Northern Ireland Assembly approves
This arrangement could, in theory, continue indefinitely, but the consent of Northern Ireland's politicians would have to be sought every four years
Customs checks on goods traded between the UK and EU would be "decentralised", with paperwork submitted electronically and only a "very small number" of physical checks
These checks should take place away from the border itself, at business premises or at "other points in the supply chain"
The No 10 source said the UK was not willing to move away from the principle of providing a consent mechanism for Northern Ireland or the plan for leaving the customs union, and if the EU did not accept those principles, "that will be that" and the plan moving forward would be an "obstructive" strategy towards Brussels.
They also accused the EU of being "willing to torpedo the Good Friday agreement" - the peace process agreed in Northern Ireland in the 1990s - by refusing to accept Mr Johnson's proposals, arguing the plan is key to respecting the so-called "principle of consent".
But Mr Varadkar has warned the Johnson plan could actually undermine that principle by giving one party in Northern Ireland a veto over what happens to the country as a whole.
It's not the official policy of the government yet...
But in government and EU circles it is becoming more likely by the hour that there will not be an agreement at next week's EU council.
There is no intention in Downing Street to move away from the broad concepts of what they are suggesting regarding either customs or the so-called principle of consent for gaining approval for the PM's plans from Northern Irish politicians.
So short of a political escape worthy of Houdini, this prime minister is moving towards making the case for leaving without a deal.
To their opponents, that might appear petulant and counter productive, but be in no doubt, if there is no deal this month, Boris Johnson's government would not suddenly play nice.
And in the likely event that there is an extension, for political reasons No 10 wants to give the impression it was forced into that position.
Read more from Laura here.
EU diplomats have told Adam Fleming there has been "zero progress" in the talks and in some areas they believe things are going backwards.
A presentation to diplomats, leaked on Monday, revealed the EU would not accept the UK's plans committing to no checks on either side of the Irish border if the Northern Ireland Assembly was granted a veto and if there was no guarantee of checks on the UK side.
Negotiators were "so nonplussed by the proposal they asked if it was a mistake", our correspondent added.
The UK's chief negotiator, David Frost, is still due to meet EU counterparts in Brussels on Tuesday, but another European official said he had so little room for manoeuvre, it called into question whether the UK was serious about getting a deal.
On Monday night, the Spectator published texts from a Downing Street source, who claimed if the deal "dies in the next few days, then it won't be revived".
The government has not denied the briefing, which also said Mr Johnson "will do all sorts of things to scupper a delay" to leaving the EU.