The Atomic Energy Organisation of Iran (AEOI) said an alarm was triggered when the woman went through screening at the Natanz uranium enrichment plant.
Officials feared she was carrying "suspicious material", it added.
There was no comment from the IAEA, which is in charge of monitoring Iranian compliance with a nuclear deal.
The announcement came as Iran rolled back another commitment under the 2015 accord by resuming enriching uranium at its underground Fordo facility. Enriched uranium can be used to make reactor fuel but also nuclear weapons.
It is the fourth such step Iran has taken in response to the sanctions reinstated by US President Donald Trump when he abandoned the nuclear deal last year.
Under the accord, Iran agreed to limit its sensitive nuclear activities and allow in international inspectors in return for the lifting of crippling economic sanctions.
Mr Trump wants to force Iran to negotiate a new agreement that would place indefinite curbs on its nuclear programme and also halt its development of ballistic missiles. But Iran has so far refused.
The other parties to the deal - the UK, France, Germany, China and Russia - have tried to keep it alive. But the sanctions have caused Iran's oil exports to collapse and the value of its currency to plummet, and sent its inflation rate soaring.
Iran nuclear crisis in 300 words
Why do the limits on Iran's uranium enrichment matter?
Iranians on Trump and the nuclear deal
The incident at Natanz seems to be the first of its kind since the nuclear deal was implemented.
In a statement, the AEOI said the inspector had been denied entry to the facility after a routine check suggested the possible presence of "suspicious material".
Iran reported the incident to the IAEA and cancelled the inspector's accreditation, it added. As a result, she left the country for Austria.
Three diplomats familiar with the IAEA's work told Reuters news agency that the inspector had had her travel documents taken, and two said she was briefly held.
The IAEA has not yet commented on the issue, which is expected to be raised at a special meeting of its 35-nation board of governors in Vienna on Thursday.
There are concerns about how the nuclear inspectors will carry out their work in future.
The board of governors meeting will also discuss Iran's reported failure to co-operate with IAEA inspectors investigating how traces of uranium were found at a site in the Turquzabad area of Tehran, where Israel has said a "secret atomic warehouse" was once located.
Under the nuclear deal, Iran is required to permit the IAEA "regular access, including daily access as reOn Monday, the head of the AEOI said it had doubled the number of advanced centrifuges being operated at Natanz. Ali Akbar Salehi told reporters it now possessed 60 IR-6 centrifuges, and that it could enrich uranium to 20% concentration "within four minutes" of being given an order.
Shortly after midnight on Thursday, Iran began injecting uranium hexafluoride into the centrifuges at Fordo under the supervision of the IAEA and the enrichment process began, state media reported.
President Hassan Rouhani said on Tuesday that Iran was aware of the "sensitivity" of the other parties to the deal regarding enrichment at Fordo, which was built in secret about 90m (300ft) under a mountain to shield it from air strikes.
French President Emmanuel Macron said on Wednesday that the resumption of enrichment at Fordo represented a "profound change" from Iran's previous position.
"I will have discussions in the coming days, including with the Iranians, and we must collectively draw the consequences," he told reporters in Beijing.
UK Foreign Secretary Dominic Raab said Iran needed to "stand by the commitments it made and urgently return to full compliance".
uested by the IAEA, to relevant buildings at Natanz".
Before 2015, Iran had two enrichment facilities - Natanz and Fordo - where uranium hexafluoride gas was fed into centrifuges to separate out the most fissile isotope, U-235.
The deal saw Iran agree to only produce low-enriched uranium, which has a 3-4% concentration of U-235, and can be used to produce fuel for nuclear power plants. Weapons-grade uranium is 90% enriched or more.
Iran also agreed to install no more than 5,060 of the oldest and least efficient centrifuges at Natanz until 2026, and not to carry out any enrichment at Fordo until 2031. The 1,044 centrifuges at Fordo were supposed to spin without gas being injected.