right Christian Democrats (CDU) of Chancellor Angela Merkel lost votes in Saxony but still came top with 32%, ahead of AfD's 27.5%.
In Brandenburg, the state surrounding Berlin, the centre-left Social Democratic Party (SPD) won with 26.2%, while AfD got 23.5%.
AfD is shunned by the other parties.
In both states the other parties will now discuss forming new coalitions - perhaps including the Greens - which will exclude AfD.
AfD's surge was biggest in Saxony, where it gained 17.8 percentage points compared with the 2014 election. The state has long been seen as an AfD stronghold.
The SPD - nationally in government with the CDU - plunged dramatically to 7.7% in Saxony.
Support for AfD grew when it campaigned against Mrs Merkel's admission of nearly a million non-EU migrants in 2015.
AfD also drew on discontent in the former communist east over Germany's closure of loss-making businesses, including coal mines.
Just how far to the right is AfD?
Angela Merkel's quiet power
AfD's slogan "let's complete the change" harked back to the 1989 "Wende" (change), which many eastern Germans see as unfinished business. Despite huge investment from the richer west, for many people the economic restructuring did not transform their lives as they had hoped.
"We're satisfied in Brandenburg as well as in Saxony," AfD co-leader Alexander Gauland said, adding that his party had "punished" Mrs Merkel's conservatives.
But despite the gains, the result may disappoint AfD as the party had hoped to come top in Brandenburg, the BBC's Damien McGuinness reports from Berlin.
The CDU state premier of Saxony, Michael Kretschmer, said "I'm very happy with the result", but added that opposition messages had made an impact on social media. "The filter bubble on the internet is so powerful, and in 20 months you cannot reach everyone," he told broadcaster ARD.
The CDU-SPD national coalition is due to last until federal elections in 2021, and a collapse could trigger a snap election or result in a minority government.
Mrs Merkel herself plans to step down as chancellor in 2021, having already resigned as CDU leader at the end of last year.