Now scientists have solved the conundrum, with the help of aerial photographs taken by drones.
Their model accurately calculated the body volume and mass of wild southern right whales. Already being used to assess the survival of calves, it has many potential uses in conservation.
Body mass is a key factor in the success of whales as a group, determining their energy uses, food requirements and growth rates.
Yet most of what we know about the body size of whales comes from old whaling literature or from animals that end up stranded on the beach or caught in fishing gear.
"It is very difficult to measure a whale on a scale - I mean you have to kill it to do it and that's exactly what we're avoiding here," said study researcher Fredrik Christiansen from the Aarhus Institute of Advanced Studies in Denmark.
The researchers studied southern right whales, which gather in large numbers at their winter breeding grounds off the coast of Argentina.
They flew a drone over whales swimming in clear water, capturing photographs when the adults and calves came up to the surface to breathe, including their backs and sides when they rolled over.
From these, they were able to obtain length, width and height measurements for 86 individuals.
They found they could get a good representation of the body shape of the whales, which they linked back to old whaling literature recording body length, girth and mass. They were then able to convert body shapes, or volumes, to mass.
"The ability to predict body mass from free-living whales opens up the opportunity for us to look at animals over time and look at how they change, how they grow," said Prof Christiansen.
Drone studies could help in conservation by monitoring the health of different whale populations in the oceans.
The approach could also be used to estimate the size of other marine mammals by adjusting the model parameters.
Baleen whales are the largest animals on this planet, ranging from the 3,500kg pygmy right whale to the 190,000kg blue whale.
Their body mass is central to their success as an animal group.
The research was carried out in collaboration with the Southern Right Whale Health Monitoring Programme in Argentina and the Woods Hole Oceanographic Institution in the US.
The work is outlined in the British Ecological Society journal Methods in Ecology and Evolution (MEE).