Netapix
Turn-key custom development of an efficient free photo editor app with many filters and effects to any taste where smart neural network turns your photo into a masterpiece in a few clicks.
- Сustom software development
- 2018 – 2020
- Industry: Photo editing, Entertainment
Initial Task
Description
Neural networks have found wide application in the field of image processing. However, high-resolution image processing algorithms using architectures such as generative adversarial network impose limitations on the execution environment because require significant computing power.
The goal of Netapix project was to develop a neural network architecture, as well as an algorithm for its training, capable of processing high-resolution images in a given style, with the ability to execute on the user’s device. This was achieved with the help of CoreML framework which was designed to work with machine learning technologies that significantly expanded the possibilities for using neural networks directly on user devices.
Technology stack
iOs
Process
Phase / 01.
Research
-
Deep neural networks showcase impressive results in the field of image processing. Our task comes down to finding a set of parameters sufficient to perform image processing transformation.
-
At the same time, each preset can be a collection of different filters, which means that there should be enough parameters to maintain variability. On the other hand, the number of parameters is limited by the amount of RAM on mobile devices and processing time.
-
We chose Convolutional neural networks. CNNs demonstrate good results in the field of image recognition, and also make it possible to significantly reduce the parameter space, which has a positive effect on processing speed.
Phase / 02.
EXPERIMENTS
-
To train the neural network, we've collected both:
- a huge data set of photos taken on the phone, and
- a set of presets created by photographers. -
The core goal of our experiments was to come up with a one-box set of parameters that would allow to perform transformation in a uniform way for each unique preset without significant loss of quality.
-
As a result of fine-tuning various training parameters as well as testing various architectures, we have managed to obtain a CNN model, which was successfully replicating all filters from an in advance prepared set with high accuracy.
Phase / 03.
Development
-
To demonstrate our custom technology, we've developed an iOS mobile application with functionality of a typical photo editor, namely:
- Camera
- Filters
- Photo editing tools
- Photo sharing -
Our mobile application refers to more than 100 pre-trained neural network models as filters, implementing the functionality of unique presets created by photographers in a uniform way.
Overall Result
As a result of our research and experiments, we can point out that performance of modern chips combined with frameworks such as Metal and CoreML significantly expand the possibilities of using machine learning algorithms directly on mobile devices. Launching an application of similar functionality with image processing in the cloud will inevitably lead to significant costs of expensive infrastructure maintenance.
At the same time, it is worth noting that neural networks seem to be a good tool for optimizing certain processes, but they could not be treated as a universal tool. For example, when adding new filters to the Netapix collection, it has come to our attention that for some combinations of effects in one preset, the trained model performed a defective transformation due to insufficient parameter space. Therefore, this optimization method should be treated as selectively universal.
