Topic
Transports
EVS worked, in collaboration with ZF Padova, on the design of a docking assistance system for sailboats based on artificial vision. The system provides the operator with a bird-eye view of the space surrounding the boat and can perceive the obstacles on the trajectory. EVS, which had already gained experience in marine vision applications, implemented an application prototype managing all phases from analysis, development, deployment to system testing.
“Our collaboration with EVS was a great experience. Thanks to their expertise in the field of augmented reality and artificial intelligence, we were able to demonstrate cutting-edge solutions for our industry. We were impressed by their approach, oriented to understand our sector expectations and to present flexible development solutions.”
― Marco Murru
System engineer for marine propulsion systems, ZF Padova
Problem
Mooring a boat is a complex operation
Mooring a boat is a complex operation even for experienced navigators. Poor visibility from the helmsman’s position, blind spots, the difficulty of seeing obstacles and perceiving their distance from the boat, make the maneuver difficult, long, and risky.
Solution
A set of cameras mounted on the mast
The solution that EVS developed uses a set of cameras mounted on the mast to observe the boat surrounding area. The system processes video streams from cameras by merging and geometrically aligning the views into a single view from the top (“bird-eye’s view”) that allows the operator to see 360° all around the boat. In addition, the system is able to perceive the presence of obstacles emerging from the water surface around the boat and indicate, in overlay, their position to the operator. This display mode provides the operator with all the information necessary to safely carry out the maneuver, similar to how we are used to when parking our cars equipped with a parking assistance system.
Added value
This system increases the maneuver safety
From the end user point of view, this system increases the maneuver safety, especially for less experienced operators or those who cannot count on the help of other crew members.
For boat builders, instead, this system may represent a chance to enlarge the user base for same vessels, including potential customers that love sailing, but have limited experience and therefore the need for assistance in delicate maneuvers.
Project partner
ZF Padova Srl, leader in the production of marine propulsion systems. It supplies systems and components for all types of ships, motorboats, high speed boats, work boats and commercial ships, to boat builders, shipyards to engine builders all over the world. It is a company belonging to the ZF Group.
Insights
The two functions of docking assistance system
The developed docking assistance system combines two functions in one:
- the generation of a 360° orthographic view of the boat by merging images from multiple cameras (“surround view”)
- the detection of obstacles around the boat indicating their presence graphically on a virtual bumper (“obstacle detection”)
The project required several computer vision skills such as the 3D reconstruction of the environment starting from a moving multi-camera system, the optical and perspective image correction, the “stitching” of the images, the brightness and color correction, the calibration of the parameters of the entire system.
One of the most challenging aspects is due to the presence of water whose appearance depends on many factors such as light, wind, environment, seabed and clearness. The water ripples, froths, gets dirty, reflects, and shines through. This sometimes makes difficult to distinguish water from obstacles. Thus, advanced machine learning techniques have been adopted to “teach” a neural network to make this distinction and to do it well even when the domain changes.
We selected the most suitable cameras and studied their optimal positioning by working in simulation and 3D modeling environments.
We chose a state-of-the-art computing platform, that was the most suitable in terms of performance, reliability, consumption, size and costs. It is a hardware module based on multicore architecture with DL acceleration and necessary multi-channel video camera support. After an initial development and validation of the algorithms in a PC environment, all the software was ported to the selected embedded platform (“embedding”), optimizing it until achieving the required performance in terms of throughput, latency, and accuracy.
The system was then tested on a real sailboat giving promising results both in terms of performance and usability in assisting the docking maneuver.
The project was carried out in close collaboration and with the support of the ZF Marine team.