Modern smartphones have become indispensable tools of everyday life, keeping track of the weather and bus timetables, helping us find our way and much more. Now you can also ‘video’ your surroundings, and the phone reads out what it sees in realtime, at a rate of 10 frames per second. This can be a huge help to people with any degree of visual impairment, from fully blind to those who only have difficulty telling colours apart.
Development has gone quickly.
“We couldn’t have done this just a couple of years ago, because phones just weren’t powerful enough,” says Simon Edwardsson in a Skype conversation with us, as he describes his company Aipoly’s app called Vision.
He is in San Francisco, and for him it’s late at night. But apparently it’s not unusual for him to be awake at that time of day. He puts in many a long day’s work. The little company he now runs with his co-founder Alberto Rizzoli is in an extremely intense phase right now.
“We’ve had a lot of media attention recently,” he says.
One reason is probably because Aipoly won the Best of Innovations Award in the category of Accessible Tech at the big CES (Consumer Electronics Show) business fair in Las Vegas earlier this year. The Vision app, which was originally only available for iPhone, has now come out in an Android version, which has triggered a new wave of buzz in the news. Soon the app will also be available in Swedish.
“It’s available in seven languages right now,” Edwardsson says. “Friends and other volunteers have translated the glossaries. First we run them through a translation app, but after that they need extensive editing, because a lot of errors creep in. ‘Hot dog’ can be translated as a dog that’s hot and so on.”
One fun anecdote he tells us is that in the first year, when Aipoly Vision was new on the market and only available in English, it became unusually popular in Japan. It was not, as it turned out, because an unusual number of blind people in Japan were well versed in English – but because the app was being used as a language teacher.
In contrast to other, similar services, Aipoly does not use cloud services to interpret the images; everything is done directly in the phone. The technology is called a ‘convolutional neural network’, and it is trained using millions of images to quickly identify an object. It’s the same technology used in self-driving cars, among other things. It’s faster than cloud services, and the user doesn’t have to deal with any possible privacy issues the cloud can create, Edwardsson explains as he illuminates what is unique about Aipoly’s app.
“For example, if a blind person takes the phone into the bathroom, they want to be sure that no one else is looking.”
Despite its success, Edwardsson doesn’t expect millions to come rolling in. There aren’t that many visually impaired people in the world, and they are often a group with small financial resources, he says.
“The app is an assistive device, so it can’t be allowed to cost too much. The simplest version is free to use. Then there’s a paid version that can do more – for example, the free version might say ‘dog’, while the paid version can recognise more dog breeds than I can.”
Instead, Aipoly is interested in collaborating with other companies that may have use for the technology behind the app. For example, it might be used to help surveillance systems interpret the data they record. “Was it a cat or a person that just passed by?”
The company is now working on a new app, a further development of Vision. It will be called Poly, and it combines object identification with providing extra data about things the user points the phone’s camera at. This is called augmented reality.
“You can get information on where you can buy a thing, what it costs and a lot of other things,” Edwardsson explains. “It’s a bit like Apple’s Siri, but image-based. Images contain so much more information, instantaneously.”
Users won’t even have to think about the advanced artificial intelligence required to make this work.
“People like my parents should be able to use the app without difficulty. It should work sort of like magic.”
But how can such a small company be so far ahead, I ask. Edwardsson says the secret to success is keeping abreast of current research.
“I try to read heaps of papers on the subject. Everything is advancing so quickly, there are new developments all the time.”
And apparently he’s quite a skilled coder – something he might thank his parents for. Edwardsson tells us that he got his first computer at age 6, but his father didn’t want him playing games on it. By age 7 he’d learned to code and was making his own games.
His passion for functional programming eventually brought him to Chalmers, and even if it was mostly random chance that he ended up in Gothenburg rather than at the Royal Institute of Technology in Stockholm, he is happy that he is here.
“After earning my bachelor’s degree, I spent a year in Japan at the Tokyo Institute of Technology, which Chalmers had an exchange with. Then I went back and finished my master’s degree.”
He did his degree project at Bombardier.
“Their security systems for trains have tonnes of different parameters to combine. It’s hard to monitor the big picture and verify that the system is behaving correctly. My project partner and I rewrote the programming language, so that what previously took a hundred lines of code now takes just two or three. Very different from what I’m doing right now.”
After earning his master’s, he went to the United States. The US Friends of Chalmers found Edwardsson an internship in Silicon Valley. The company he worked for analyses sales data via the cash registers in big chains like Walmart and Target. They look at several variables affecting sales.
“That job taught me more about machine learning.”
When his work visa expired, Edwardsson moved to Shanghai and continued working there for the same company. Later he moved to the UK and became the head of the company’s research development. At the same time, he began working on Aipoly, which has now become a full-time job – even beyond full-time.
“I just couldn’t have two jobs anymore,” Edwardsson says.
And he hasn’t decided yet whether he’ll choose to read a scientific article when we hang up, or just go to bed. As he said, to succeed in the app branch, a company has to stay on top of every development.
Text: Siri Reuterstrand