The gunman was shot dead by police shortly after he began firing, although police are investigating reports that a second suspect may still be at large.
The Gilroy Garlic Festival was about to end for the weekend on Sunday evening, when shots were fired at the site.
There was "a white man in his early to mid-30s firing a rifle," eyewitness Julissa Contreras told NBC.
It was the 246th mass shooting in the US this year, according to US tracking website Gun Violence Archive.
The suspect entered the festival after cutting through a perimeter fence, Gilroy Chief of Police Scot Smithee told reporters. He said witnesses reported that a second suspect may have been involved, possibly in a support role.
Officers were already at the site and responded to the shooting in less than a minute, Chief Smithee added.
Videos posted on social media showed people running away from the festival, which took place 30 miles (48km) south of San Jose.
"What's going on?" a woman can be heard asking in one video, "who'd shoot up a garlic festival?"
"We couldn't feel worse," Brian Bowe, the festival's executive director, said at a news conference on Sunday. "It's just a horrible thing to experience."
"This is nothing short of horrific," tweeted California Governor Gavin Newsom.
US President Donald Trump also tweeted about the shooting, urging people to "be careful and safe".
Some people are being treated for critical injuries, a spokeswoman for the Santa Clara Valley Public Health System said.
The Santa Clara County Medical Center has so far admitted five victims, CBS reports. Two patients are being treated at a medical centre at Stanford University, according to CNN.
Michael Paz, 72, a hat vendor at the festival, told the San Francisco Chronicle that he saw a gunman armed with a rapid-fire assault rifle.
"He came ready to shoot because he was wearing a protective vest," Paz said. "He was shooting left; he was shooting right without any particular aim."
As police swooped on the gunman and shots were fired, festival attendees dropped to the ground, Mr Paz said.
"It was just rapid firing," Ms Contreras told NBC Bay Area. "I could see him shooting in just every direction. He wasn't aiming at anyone specifically.
"It was just left to right, right to left. He definitely was prepared for what he was doing."
Evenny Reyes, 13, was among those who scattered in confusion when gunfire erupted at around 17:30 local time (00:30 GMT Monday).
"We were just leaving and we saw a guy with a bandana wrapped around his leg because he got shot," she told the San Jose Mercury News.
Christmas Hill Park, where the event was taking place, bans weapons of any kind, according to the festival's website.
Gun Violence Archive defines a mass shooting as a gun attack in which at least four people are either killed or wounded.
In total, it estimates that 8,434 people have died as a result of gun violence in the US in 2019 so far.