Company of the Month: Fizyr

Company of the Month: Fizyr

Every month we select one company which represents a positive example of innovation in logistics and has the potential to alter the way the industry operates. This month, Transmetrics selected Fizyr, the developer of deep learning for vision-guided robotics, as the June “Company of the Month” for its exceptional approach to the advancement of robotics technologies. In order to learn more about the company and what they do, we interviewed Herbert ten Have, CEO of Fizyr, and asked him several questions about the company and the industry.


First of all, congratulations on becoming the Company of the Month! Can we start from briefly introducing our readers to what Fizyr does? How does your product work?

We are a deep tech company, based in RoboValley Delft. We are automating logistics globally and robustly putting robots to work. Our deep learning algorithm adds a layer of understanding, bringing autonomous decision making to processes that involve identifying, quality control, counting, picking and manipulating goods. Fizyr develops world’s best-trained algorithms to enable robots handling any pile of various items or fully unknown parcels using machine vision.

So what we do is taking a 3D color image, so it defines RGB colors and depth for the algorithm and then we perform the following 4 steps:

  1. Segmentation is the most important one. Boxes are often aligned right next to each other and they have to be segmented, otherwise, they will be overlapping on the image and it’s grabbing either 2 boxes, 1 or none.
  2. Classification – our algorithm classifies an item as a box or a plastic bag or textile or a parcel.
  3. Quality Control – it is not always done, but we are currently doing it more and more. We teach our algorithm to find unknown defects. For instance, when boxes or parcels are being sorted, warehouse operators don’t want open boxes or open bags to go into the sorter, because it might damage it. Through classification and quality control, our algorithm can determine if the conditions of the object are too bad to be put into the sorter or to pick it up at all.
  4. Finding 6 Degrees of Freedom1Six degrees of freedom (6DOF) refers to the specific number of axes that a rigid body is able to freely move in three-dimensional space. It defines the number of independent parameters that define the configuration of a mechanical system. Specifically, the body can move in three dimensions, on the X, Y and Z axes, as well as change orientation between those axes though rotation usually called pitch, yaw and roll. Grasp Location is the final step. What does it mean? It means that we can determine the grasp location at many possible angles. So, we can find the proper grasp location for a lot of different objects and it does not matter what the position of an item is.


The whole process is done for all the elements in the picture and it takes less than 0.2 seconds to perform all the 4 steps. We use a GPU (Graphical Processing Unit) for that, which is responsible for the neural network processing. And it is all built in the Linux based PC, which is standing next to the robot and our algorithm takes the images from the camera and within a couple of milliseconds, it defines all the needed data.

What kind of technologies allowed Fizyr become a reality? How much did you develop the algorithm over the past several years?

“The open source community believes that our algorithm shows much better results than Facebook does. We are faster, better, more accurate and we have a very robust code!”

Briefly speaking, our main technologies are Artificial Intelligence for machine vision and Deep Learning. In order to create the algorithm we started with existing research from technical universities, and we also use open source tools like Tensorflow. The Keras-RetinaNet implementation we made to generate algorithms is also put as an open source on Github and we are extremely proud of that. Some other companies such as Facebook do it as well, and the open source community believes that our algorithm shows much better results than Facebook does. We are faster, better, more accurate and we have a very robust code!

And to point out the pace of our progression, 4 months ago we did something for a postal company, and now the result is twice as fast and that’s an incredible speed of development. We have huge datasets for which we apply supervised learning. One of our assets is the millions of annotated pictures we can use to train the algorithms so we educate the robot to be robust in a particular environment.

Based on your experience with logistics businesses, what are the top 3 trends that are changing the industry at the moment and why they are important?

I would say that the three trends are artificial intelligence, Goods to Men Systems (GtMS), specifically Automated Guided Vehicle (AGV’s) and the third one being the shortage of people who want to work in logistics, particularly in the warehousing.

Let’s look at the current average warehouse – 55-60% of the cost is going to people, operators. Thus all the picking worldwide, wherever you go, is done by humans. Even in Amazon, or in any other big company, you name it, the picking itself is still done by humans. There are maybe a few little exceptions, but let’s say 99% it is all done by humans because it is very hard to train robots to cope with variation. Coming back to Amazon – they have 20+ million different products, and it is almost impossible to train the robot to find the grasp locations for all of these different items. With Deep Learning algorithms we can teach robots to pick up a wide variety of objects. Actually, recently we trained the algorithm to pick up completely unknown objects. There are currently a lot of companies that are doing the bin picking and they use a CAD model of an object they are trying to grasp so it is relatively easy and has been around for many years. You just need a good camera and a good model and then you can pick, so that is not an issue. On the contrary, the differentiation in shape, in size or in color, or in case the object is completely random makes this process much harder, because the models do not work. At Fizyr, we have proven that we trained our network so we can handle this variation.

As for the GtMS, there are a lot of them and they are important and already widespread. Take all the big integrators in Logistics, they all create GtMS and it could be a Conveyor Belt, AGV or a Shuttle. Basically, any device that brings goods to the human order picker could potentially be automated now if the robot and gripper can handle it. A lot of companies are investing right now in AGV’s in particular because it decreases the involvement of humans and increases the efficiency of warehouse operations.

The third trend, however, is not on the technology side, but it is on the human side as I said. There is a shortage of people who want to work in logistics. And we see that it is getting worse because in logistics people are often working late at night or early in the morning. And people simply don’t want to do that work anymore. This is not the matter of cost, but it is a matter of business – “Can we get people?”, “Can we get the work done?” We see that in the Netherlands, in Germany, in the UK and even in the US. So there is a shortage of people and it is a serious and worrying trend. However, that is the additional motivation for the logistics players to start the automation of their processes.

Can you please tell us more about the team of Fizyr? Why do you think, the challenge of machine vision for handling is interesting for people in your company?

Currently, we have an international team consisting of 13 people. We have fun, act as friends and are very committed to staying #1 in the world of object detection. We believe that we create awesome things with the technology in our hands!

We are doing something that has never been done before. One of the things we have done is we can pick a towel from a pile consisting of 100+ towels from the washing/drying machine. And we were able to find corners of the towels. We’ve also done things in horticulture and agriculture that nobody has done before. Apart from logistics, sometimes we do side projects to test the technology and that’s really awesome. We have also met a lot of enthusiasm from our clients that are quite often huge international companies. Our clients provide us with challenges, ask us if we can solve it and when we solve it makes everyone on the team very proud.

Picture Credit: Fizyr

You are a double winner of an Amazon picking challenge. Can you please elaborate on the challenge and how did you manage to win?

To be completely clear, we won both the picking challenge and the storing challenge. At that time we were called Delft Robotics, and we were still a robotics integrator, therefore we did both the hardware and the software part. In the picking challenge, we had a number of items from the shelf that we knew on the forehand and you could pre-train the algorithm in order to recognize them on the shelf. However, one of the difficult parts is that shelves are made for humans and not for robots. Secondly, it is very hard to identify the particular object out of the variety of objects. Thirdly, the robot has to grasp them and there is no gripper so flexible as human hands, so we had to think about how to solve that problem as well. What made a difference for us is that our team applied deep learning to solve this problem and that’s how we won. The same was done for the storing challenge but in the reverse order – we had to put items back on the right shelf. However, that was a long time ago and now our technology has developed much further.

If it is not a secret, what are the next big goals for Fizyr in 2018?

We doubled the team last year. I would love to do it again to cope with the expectations of our clients. We are still bootstrapped and we spend most of our time working for clients earning money, so we currently grow steadily. However, we want to speed up, and grow faster than before and therefore we want to raise money, so we are aiming for series-A quite soon and we are preparing for that. The big goal for this year is to receive and use the external money to really boost our growth.

From June 19th-22nd Fizyr has an impressive demo on Automatica in Munich, Hall B6.502.
Sneak preview: https://www.youtube.com/watch?v=8B1sbf1x-6Q