If you are anything like me, one of the things you love about this digital era is that you can be artistic and creative, even if your drawing skills never made it
much past stick figures.
The average smartphone can take photographs of
astonishing quality. And all sorts of special effects, filters, and overlays enable users to convert their favourite photos into unique expressions of themselves.But all of this still generally remains within the realm of photographic content.
What if there was an algorithm so smart, that it could actually analyze your favourite work of art and then 'paint' you a replica in the same style? Not some pixel-by-pixel analysis, but evaluating it as a talented artist would:
taking into account colors, brush styles and other artistic techniques and then applying those styles in a way that blends with whatever target content you choose.
In fine art, especially painting, humans have mastered the skill to create unique visual experiences through composing a complex interplay between the content and style of an image.
The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general.
One of the most interesting discussions today around within machine learning is how it might impact and shape our cultural and artistic production in the next decades. Central to this discussion is the recent advances in
image style transfer using deep learning.
“To use Machine Learning specifically the technique of artistic style transfer to aid in the creative process.”
What is artistic Style Transfer?
One of the most exciting developments in deep learning to come out recently is artistic style transfer, or the ability to create a new image, known as a pastiche, based on two input images: one representing the artistic style and one representing the content.
style transfer with deep networks
Style transfer can be defined as finding a pastiche image p whose content is similar to that of a content image c but whose style is similar to that of a style image s . This objective is by nature vaguely defined, because
similarity in content and style are themselves vaguely defined.
The neural algorithm of artistic style proposes the following definitions:
• Two images are similar in content if their high-level features as extracted by a trained classifier are close in Euclidian distance.
• Two images are similar in style if their low-level features as extracted by a trained classifier share the same statistics or, more concretely, if the difference between the features’ Gram matrices has a small Frobenius
My experimentation with style transfer
Post reading all the research papers and going through all the tutorials I started working on creating my own Style Transfer App.
N-STYLES TRANSFER NETWORK
The technique of Style Transfer so far was focused on the transfer of the style from a single style image s to a content image c to create a Pastiche.
But what if we can start merging styles from different paintings together into a single content image to create a varied patische?
This stems from the intuition that many styles probably share some degree of computation, and that this sharing is thrown away by training N-networks from scratch when building an N -styles style transfer system. For instance, many impressionist paintings share similar paint strokes but differ in the color palette being used. In that case, it seems very wasteful to treat a set of N impressionist paintings as completely separate styles.
What's in this app?
This app is an Android app that takes frames from the device's camera and renders them to a view on the main activity.
• The first button, labelled with a number (256 by default) controls the size of the image to display (and eventually run through the style transfer network). Smaller numbers mean smaller images, which will be faster to transform, but will be lower quality. Conversely, bigger images will contain more detail but will take longer to transform.
• The second button, labelled save, will save the current frame to your device for you to use later.
• The thumbnails represent possible styles you can use to transform the camera feed. Each image is a slider and you can combine multiple sliders that will represent the ratios of each style you wish to apply to your camera frames. These ratios, along with the camera frame, represent the inputs into the network.
about this network
The network we are importing is a result of a number of important developments. The first neural style transfer paper (Gatys, et al. 2015) introduced a technique that exploits properties of convolutional image classification networks, where lower layers identify simple edges and shapes (components of style), and higher levels identify more complex content, to generate a pastiche. This technique works on any two images but is slow to execute.
A number of improvements have since been proposed, including one that makes a trade-off by pre-training networks for each style (Johnson, et al. 2016), resulting in real-time image generation.
Finally, the network we use in this lab (Dumoulin, et al. 2016) intuited that different networks representing different styles would likely be duplicating a lot of information, and proposed a single network trained on multiple styles.
An interesting side-effect of this was the ability to combine styles, which we are using here.
inside the network
This network was generated using Tensorflow. Before using it in an environment with constrained resources, such as a mobile app, this model was exported and transformed to use smaller data types & remove redundant calculations.
The end result is the stylize_quantized.pb file, displayed below, that I use in the app.
examples of pastiche created using the app
Below are some examples of Pastiche created using the APP.
THE NEXT STEP
Now that I have been able to create a way to merge styles from various paintings together and allow for the user to have full freedom over the final output Patische they create, I feel this technology can be used by two very
1) The Artist/Designer
2) The Consumer (general public)
Below is a Persona Mapping and possible use cases.
THE concept video
*This project was built using Google's Android Experiments instructions.