The protesters, who demanded that Iran stop interfering in Iraq's internal affairs, climbed the consulate's walls.
There were reports that the security forces opened fire, but another account said unknown gunmen were responsible.
It came as mass demonstrations against Iraq's government continued in Baghdad despite a plea from the prime minister.
Adel Abdul Mahdi called on the protesters - who are angry at his failure to deal with widespread corruption, high unemployment and poor public services - to re-open roads and "help restore normality to the country".
Iraq protests: What's behind the anger?
Is a new Arab Spring unfolding in the Middle East?
He said the government respected peaceful protests, but that the blocking of roads around the capital and key ports in the south, combined with the "threatening of oil interests", had cost the country billions and contributed to price rises.
President Barham Saleh said last week that Mr Abdul Mahdi was willing to resign once a replacement was found, and that a new election law was being drafted.
But the protesters want to sweep away the entire political establishment.
Iran, which backs the Iraqi government and a number of powerful local Shia militias, has urged protesters to seek changes within the "framework of legal structures", warning them that "the enemy wants to break" the structures.
"The US and Western intelligence services, with the financial backing of reactionary countries in the region, are spreading turmoil," Iranian Supreme Leader Ayatollah Ali Khamenei said last week.
On Sunday, dozens of people shouting "Karbala is free, Iran out, out!" took part in the protest outside the Iranian consulate.
Some of the protesters threw stones and burning objects over the high concrete walls surrounding the building. Three men scaled the walls and waved Iraqi flags.
Correspondents from the AFP news agency at the scene reported that security forces fired live ammunition to disperse the protesters, and that several were shot. The agency later cited a medical official as saying that three protesters succumbed to their wounds.
The Iraqi High Commission for Human Rights (IHCHR), an official watchdog, said three protesters were killed in a clash outside the consulate.
But a source in Karbala told the BBC that one of the dead was a guard at a nearby Shia shrine who happened to be passing by. The source also said the origin of the gunfire was unknown and that it targeted both the protesters and security forces.
Iran's consul general in Karbala, Mir-Masoud Hosseinian, told Iranian media on Monday morning that the situation was under control and that "conditions around our country's diplomatic residence have returned to normal".
The IHCHR also reported that Saba al-Mahdawi, an activist and doctor who had been providing first aid at the protests in Baghdad, was abducted by unknown men on Saturday night. It called on the authorities to investigate her disappearance.
More than 250 people have been killed and 9,000 injured in clashes with security forces in two waves of protests since 1 October.