Driven by a shortage of skilled workers and increasing wages worldwide, robotics researchers and horticultural growers are partnering to develop robots that enhance the productivity of the human workforce.
Discerning robotic vision is key when designing a picking or harvesting robot. Improvements in machine-vision algorithms and the use of multi-spectral cameras are helping the latest robots to find their targets in a crop much faster.
Early bots designed at the Dutch horticultural research facility at Wageningen University chose to start with yellow peppers, what one could call the ‘low hanging fruit’.
Teaching a camera on a robot to find yellow or red peppers in a leafy green crop is relatively simple compared to the task that Farshid Amirabdollahian, professor of human-robot interaction at the University of Hertfordshire, set himself.
Working with one of the largest growers of cucumbers in the UK, his robots had to learn to find a green fruit often partially masked by green leaves.
He equipped the robot with a stereo camera that is also sensitive to infrared light, which provides more textural information, helping the robot to differentiate between fruit and leaf. “Training the machine-learning algorithm is painstaking and laborious, involving hand-?labelling thousands of images,” he says.
Considering that skilled humans can pick a cucumber every 3-4 seconds, it would seem that the human workforce has little to worry about, but Amirabdollahian and his commercial grower partner, Glinwell (a supplier to Tesco), calculate that a robot has only to get down to 10 seconds per fruit to compete, as it will be able to work for 16 hours a day or more.
It is also possible that robots could work slowly through the night.
A future cucumber robot could even be linked into the environmental control system in the glasshouse that delivers nutrition to the growing plants, and ensures that the environment is pest-free, and optimal for growth, by controlling ventilation and shading when necessary.
The international strawberry crop has been the focus of Dogtooth Technologies, a spin-off from Cambridge University that finds itself at the forefront of robotic berry harvesting, with tens of now generation-3 robots deployed in Kent in the British summer, and in Australia during the UK winter.
“Picking delicate fruits such as strawberries and raspberries requires visual acuity, dexterity and the ability to work in an unstructured environment,” says co-founder and CEO Duncan Robertson.
“Berries are suspended on complex stem structures and surrounded by foliage. A picker robot must make sense of the complexity in order to find and pick the fruit. The cutting point on the stem may be only 2mm across, and must be located precisely by the robotic picking arm to harvest without causing damage.”
Dogtooth has developed a cutting-edge machine-vision system for robotic control which is the subject of a patent application, but Robertson is keen to point out that a large part of the challenge is more pedestrian – building robots that can work robustly and reliably outside in rain and mud, for many hours a day.
Considering that its complex robots have as many parts as a family car, and that they have only been in development for five years, that is quite an achievement. “Our robots are built like tanks, to handle any terrain,” he says. Robertson believes that it is inevitable that the bulk of the global berry crop will be picked by robots before the end of this decade.
A human strawberry-picker costs 50p to pick a kilo of fruit and Dogtooth’s robots have to pick at a comparable cost to be competitive. But the robots deliver other benefits – consistency of grading and quality control, and overnight picking when the temperatures are lower, resulting in fruit with a longer shelf life.
And as they move through the crop, they can also gather data that is valuable to the grower – monitoring and mapping the as-yet unripe berries, to predict future yield, helping growers to negotiate better prices.
Focusing on improving the productivity of the existing workforce is Marc Hanheide, a professor of intelligent robotics and interactive systems at the University of Lincoln. He runs the RASBerry (Robotics and Autonomous Systems for Berry production) project there.
“It’s all very well for the politicians to say ‘bring in the robots’, but we’re not quite there yet. I can see a time where robots might pick 80 per cent of the crop and human pickers go in to take in the remaining 20 per cent. It’s a very different operational model for growers. Robots can make the existing human pickers that much more efficient,” he says.
Hanheide works in collaboration with commercialisation partner Saga Robotics, a Norwegian field robotics company. Fruit pickers spend up to 20 per cent of their time moving trays of berries to the refrigerated warehouses on the farm. That has to be done repeatedly and quickly to retain a good supermarket shelf life for the fruits, and is an ideal task for a robot.
Now, aided by advanced cameras, drones and sensors, technology will screen large numbers of plants automatically, measuring characteristics such as leaf shape, plant architecture and physiology
Trays left out in the sun in the field deteriorate fast. One hour in the sun equates to one day less shelf life. “When there are crates ready to go to the refrigeration unit, the picker will call in the robot, which is a modular, wheeled design called Thorvald that can be equipped with various sensors, cameras and arms according to its task on the farm.
We have to anticipate the interactions between the humans and the robots in the field. They can view the robot as a competitor, can sabotage it, or even manipulate it for their own gain,” adds Hanheide.
Hanheide’s team were also responsible for the navigation system on three earlier versions of Thorvald which are already hard at work in several nurseries in Kent cruising up and down the rows of plants at night. This version uses ultraviolet light sterilisation to eliminate mildew – a fungal disease that can rot the fruits. The UV light treatment reduces the need to apply chemicals to the crop.
In 2019, seasonal labour shortages resulted in thousands of tonnes of fruit and veg being left to rot, unpicked in the fields, according to the trade magazine The Grocer.
Yes, labour shortages are a compelling reason to develop horticultural robots, but they are not the only reason. Tackling the harvest from a different angle, plant breeders have changed tack from producing varieties of our favourite fruit and veg that are easy for humans to pick, and are incorporating qualities in future varieties that will be easier for both robots and humans to pick.
Adam Whitehouse is the head strawberry breeder at NIAB EMR (the venerable East Malling fruit research establishment in Kent which is largely funded by the growers) and has decades of experience in producing top varieties. “It usually takes eight years to produce a new strawberry for the industry, through screening over 18,000 seedlings, and then a number of years of trialling before the new variety is released. We have to anticipate the future needs of the industry,” he says.
Changes to the climate will be important, but also the structure of the plant – how the berries present themselves to pickers, robotic or human. Larger, well-displayed berries mean faster picking, but fruit that ripens evenly will decrease the time required for pickers to recognise them and pick over the crop. Whitehouse released Malling Centenary, which quickly became established as the industry standard variety.
Skilled plant breeders like Whitehouse will never be out of a job, but technology will certainly be deployed to help him become even more effective. The phenotype of a plant – how it looks and performs – is the result of the interaction between its genes and the environment in which it is grown. Plant breeders have been assessing the phenotypes of plants for centuries, picking the best performers and creating more floriferous decorative plants or new higher yielding vegetables.
Now, aided by advanced cameras, drones and sensors, technology will screen large numbers of plants automatically, measuring characteristics such as leaf shape, plant architecture and physiology. And if genetic samples are taken from each seedling at the beginning of its life, its performance in the real world can be increasingly correlated to a plant’s genetics directly. Fast and objective automated data collection from many thousands of seedlings throughout their growing life will save time and money.
Poor performers can be eliminated at an early stage. This will both cut down the time needed to produce new varieties with better yields or resistance to diseases, and also increase the selection intensity through observing and screening much greater numbers of plants.
At King’s College, London, robotics and machine learning engineer Matthew Howard’s team are working with a Chichester producer of another high-value crop, cut fresh culinary herbs, including basil, coriander and parsley. Some of these are imported from overseas in large bales, while others are grown in the UK according to the time of year. Currently, low-paid immigrant workers have to stand for hours in refrigerated rooms parcelling up the herbs into 15g bunches for sale in supermarkets.
We are probably some decades away from dispensing with the human workforce in horticultural crops altogether, but task by task, automation is transforming the business
“We have been using machine vision and machine learning to replicate the strategies used by the human pickers,” Howard explains. “Using a two-finger gripper and cheap 3D RGB cameras repurposed from gaming, we can produce a depth image or map that allows our prototype robot to grab 15g bunches of rocket and parsley with good consistency. Each herb presents a different challenge for machine vision as the plant architecture is different.”
In the pack house of the future, robots may work without light, and at lower temperatures than humans could tolerate, which would boost the shelf life of the herbs.
Another high-value crop is cut flowers, and the Dutch are at the forefront of applied robotics in this sector in their vast glasshouses and tulip fields. There are hundreds of robots at work in Dutch horticulture. In one giant glasshouse in Waalzicht, the size of eight rugby pitches, three robots developed by ISO Group have planted almost 100 million young plants since they began work in February 2018.
Before the introduction of the robots, young, mostly Polish, workers would lie prone inches from the soil in 32°C heat and 100 per cent humidity, for hours at a time to plant cuttings of lisianthus, a popular cut flower. Now, the precisely repeating actions of the lisianthus planting robot at Waalzicht nurseries is mesmerising, and contrasts with those of its human caretaker, a gangly lad who seems clumsy in comparison. His job is to keep three robots supplied with a constant stock of trays of tiny young plants, keeping an eye on their output and correcting the few mistakes in planting that they make.
Although the process is partly automated, humans must still uproot the flowering plants, gather them into bunches, pass them on to a conveyor belt, trim the roots, cut the stems and insert the flowers into sleeves ready for sale.
We are probably some decades away from dispensing with the human workforce in horticultural crops altogether, but task by task, automation is transforming the business.