A Digest of Deep Learning Pearls

All you need is time and GPU

Try to allocate time for these thought provoking Deep Learning papers. Part of them with try it yourself implementation at GitHub.

1. Try it yourself at home or anywhere at all (with GPU)

Transformer more than meet the eye!
– A novel approach to language understanding from Google Brain(via David Ha)
It is a very interesting solution for an old linguistic/ syntactic challenge (anaphora) with Deep Learning. More detailed explanation of anaphora resolution.
– Based on “Attention is all you needpaper

2. Learning To Remember Rare Events

An interesting approach to introduce memory module into various types of Deep Learning architectures to provide them with life long learning.

3. One Model To Learn Them All

A unified Deep Learning model that is capable of being applied to inputs from various modalities. It is a one step closer toward general DL architectures.

4. Meet Fashion-MNIST

Finally, it is time to ditch MNIST in favor of Fashion-MNIST

Which is better from a number of aspects. Which one? Find yourself.


If you haven’t noticed the one thing in common to all of these items except for one is
Łukasz Kaiser researcher from Google Brain.

 Java Code Geeks


NLP is Natural Language Processing

Get ready for a real NLP

I am back to blogging and have a motivation to post a number of posts (or at least one) on the subject of Natural Language Processing. Upcoming posts also will contain information on recurrent neural networks such as LSTM. So stay tuned.

For now, check this out

If you are into Natural Language Processing (NLP) then you may find links below useful.


1. Attention Is All You Need paper in arxiv.



No winter but AI global warming


Name things for what they are

Is Deep Learning rage simply a bubble or is this time it here for a long time to stay. As researchers proposed first let’s change the Deep Learning title into the more humble and exact Multilayered Network for Functions Approximation. Now it sounds more practical and there is no sign of hype. Then check to what fields those networks were applied and see if it is diverse and if the algorithms used are universally applicable. Check the number of articles published that have a real essence within them. If you’ve got ‘yes’ as an answer to those questions then it feels like finally those approaches are really usefull.

What’s next?

This post will be updated in a near future. Meanwhile check the posts by Carlos E. Perez from IntuitionMachine.com that writes extensively on the subject and do not forget to check his ‘The Deep Learning Playbook

Wind of Deep Change

Welcome to the world of Machine and Deep Learning

Following my transition to another continent in near future I’ll be able to focus more on Machine and Deep Learning being a technical editor at renowned Machine Learning Mastery site authored by Dr. Jason Brownlee. It means you can expect more posts on machine learning to come especially on LSTM and recurrent neural networks.

What is it like to be a technical editor?

Throughout my career I’ve been a SW test engineer and SW developer but in parallel I’ve been busy helping to edit books as Jumping Into C++ by Alex Allain and other projects, such as Kindle Optimizer Chrome extension. So becoming a technical editor in machine learning field is just a logical step to make. Actually technical editor is a bit like a QA engineer and a developer at once since you have to understand how Python code is working to make that LSTM to be able to predict time series values and to be a test engineer to make the content and the code to be as good as it can be. In addition, there is a kind of freedom that regular tester or developer do not possess which is to suggest changes to the author which may be meaningful and influential. Most importantly, technical editor deals with the raw content of a future article, a blog post or a chapter from the book that millions of people may read and it provides you with the understanding of the responsibility that you bear on your shoulders. The corrections that you make may influence readers and make their experience pleasant or not.

Why machine or deep learning after all?

Technical editing as testing or programming is a universal position since it can be successfully applied to various topics in those fields, but machine learning has the proper ingredients of math, programming and future potential that makes it very attractive.

Stay tuned as John Sonmez says

So if you follow this blog stay around the corner to be up to date with the current progress in Deep Learning field and if you care check this public Deep Learning for All group at Facebook where I share latest and in my view greatest news coming from Deep Learning fruitful field.



OpenCV installation on Linux and Windows


How hard is to install OpenCV?

This was the question that I asked myself lately when I needed to use OpenCV for a project. I thought it must be simpler on Ubuntu than on Windows. But I was wrong. The goal of this tutorial is to provide working guidelines for OpenCV installation. I’ll cover installation instructions for OpenCV with following configurations:

Windows 7/ 10 

  • OpenCV 3.x.x with Python 2.7
  • OpenCV 3.x.x with Python 3.5

Ubuntu 16.04 

  • OpenCV 3.x.x with Python 3.5

Installation on Windows 7/ 10

OpenCV 3.x.x with Python 2.7 on Windows 32 bit

To have all the dependencies that are related to Python it is useful to install Anaconda.

  • Install Anaconda 2 for Python 2.7 (32 or 64 bit)
  • Install Anaconda 3 for Python 3.5 (32 or 64 bit)
  • Now we can install OpenCV by using pre-built libraries by downloading them from here.
  • For the sake of this tutorial I used OpenCV version 3.2.0
    • opencv-3.2.0-vc14.exe
  • After you’ve installed downloaded OpenCV version there is a need to move cv2.pyd file to a Python installation library.

Look for the cv2.pyd at the opencv installation folder


And move the cv2.pyd file to Python 2.7 installation folder



Example application

  • To test that opencv installed correctly
  • Open command line and run python. Then type the commands below to figure out what is the current opencv version.
Python 2.7.13 |Anaconda 4.3.0 (32-bit)| (default, Dec 19 2016, 13:36:02) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cv2
>>> print(cv2.__version__)

OpenCV 3.x.x with Python 3.5 using Wheel on Windows 7 64 bit

  • There is no library for Python 3.5 support in OpenCV out of the box that is why we can use  unofficial Windows binaries for Python extension packages from here to be able to use it.

Note: I downloaded this one because I have Windows 7 64 bit

  • opencv_python-3.2.0-cp35-cp35m-win_amd64.whl

Pay attention that 3.2.0 means opencv version i.e. opencv-3.2.0

cp35 means Python version i.e. Python 3.5

  • After you downloaded this file open the command line and open the directory this file located in. For example, let’s say it was downloaded to Downloads folder.
  • Change current folder to Downloads 
C:\>cd C:\Users\You\Downloads
  • Install wheel with pip install command
C:\Users\You\Downloads>pip install opencv_python-3.2.0-cp35-cp35m-win_amd64.whl
Processing c:\users\andrei\downloads\opencv_python-3.2.0-cp35-cp35m-win_amd64.whl
Installing collected packages: opencv-python
Successfully installed opencv-python-3.2.0
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

  • Pay attention that you saw this line ‘Successfully installed opencv-python-3.2.0’

Example application

  • To test that opencv installed correctly
  • Open command line and run python. Then type the commands below to figure out what is the current opencv version.
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print(cv2.__version__)

Additional resources

Installation on Ubuntu 16.04

To install OpenCV on Ubuntu follow the steps in the guides below. The first one is the best and it worked for me.

  • Simply run this command for basic opencv3 installation.
conda install -c menpo opencv3
  • If Anaconda is not installed then run this one to install it.
sudo apt-get install python-opencv

Additional resources

What’s next?

Now that you have a working OpenCV you may watch this nice tutorial by Siraj Raval that is funny and hands on with OpenCV. It will teach you How to do Object Detection with OpenCV. It will also teach you that there is a need to run a code at least once before filming a YouTube video.

In addition if you are interested in object detection with OpenCV then definitely look at Satya Mallick tutorial on the subject.

 Java Code Geeks

Kids gonna love LSTM deep learning network


Prepare for an upcoming Android game from neaapps applications development. This game will be based on a LSTM deep learning network for prediction of a next character from various length characters string.

For now you can check out the already existing apps brought to you by neaapps.

Why Long Short Term Memory deep learning network?

It turns out that LSTM is very good at learning and predicting sequences of patterns. That is why it is natural to use it for creating engaging games for little and not so kids. For more information what is LSTM and how to use it read Chris Olah’s post.

Stay tuned

It will be available soon in the nearest  Google Play Store.


The inspiration for this application came from a chapter on LSTM from Jason Brownlee’s Deep Learning With Python book.

Multilayer Perceptron Predictions Exposed

start_mlp_image.pngLearning Deep Learning

Currently, I learn Deep Learning fundamentals with the help of Jason Brownlee’s Deep Learning with Python book. It provides good practical coverage of building various types of deep learning networks such CNN, RNN etc. Running each model in the book is accompanied with providing various metrics, for instance, accuracy of the model. But accuracy does not provide a real feeling of the image recognition. To improve upon this I updated one of the code samples that came with the book with my own implementation that ran the model with an image file to perform a classification on it. The detailed steps explaining this follows.

What Will We Do?

This tutorial explains how to build a working simple multilayer perceptron network consisting of one hidden layer. In addition to working model that is trained on handwritten digits of MNIST data-set we’ll see how can an image of a digit taken from this data-set can be classified using this network.

  • Multilayer perceptron network is composed of a network of layers of fully connected artificial neurons. In our case it is built of three layers: input layer, hidden layer and output layer.
  • MNIST database is an abbreviation of the Mixed National Institute of Standards and Technology database for handwritten digits. It has a training set of 60,000 examples, and a test set of 10,000 examples. It is used for supervised learning of artificial neural networks to classify handwritten digits.

How Do We Do It?

To accomplish this task there is a need to fulfill certain prerequisites. A number of steps below were explained in a post on Keras, Theano and TensorFlow (KTT).


  1. Supported operating systems are 
    1. Ubuntu 16.04 64 bit
    2. Windows 10 or 7 64 bit
  2. Python 2 or Python 3 installed with Anaconda 2 or 3 respectively. See KTT for more details.
  3. Works with following deep learning libraries. See KTT for more details.
    1. TensorFlow and Theano
    2. Keras
  4. May be run within Jupyter Notebook. See installation steps here.

Building a Network

The multilayer perceptron in this particular case is built of three layers.

  • Input layer with 784 inputs that are calculated from 28 x 28 pixel image that is 784 pixels.
  • Hidden middle layer with 784 neurons and rectifier activation function
  • Output layer with 10 outputs that give a probability of prediction by softmax activation function for each digit from ‘0’ to ‘9’.

The overall structure of the network


The Code Overview

The code structured as  follows.

  1. Import proper Python libraries for working with Deep Learning networks.
  2. Fetch image file from a disk in accordance with host Operating System
  3. Load an image to be classified 
  4. Load MNIST dataset and preprocess images pixels into arrays
  5. Define helper functions 
  6. Prepare multilayer perceptron model and compile it
  7. Check if trained model exists
  8. If not train new model, save it and predict image 
  9. Else load current model and predict image 

The code

The code below is brought to you in full and can be found in GitHub repository in addition to saved model and Jupyter Notebook that makes it possible to run this code module after module in a really interactive way.

  • Import proper Python libraries for working with Deep Learning networks
# Baseline MLP for MNIST dataset
import numpy
import skimage.io as io 
import os 
import platform
import getpass
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.utils import np_utils
from keras.models import model_from_json
from os.path import isfile, join

# fix random seed for reproducibility
seed = 7
  • Fetch image file from a disk in accordance with host Operating System. In our case it is a 28 x 28 pixel image of ‘3’ digit form MNIST dataset
  • Load an image to be classified 
# load data
platform = platform.system()
currentUser = getpass.getuser()
currentDirectory = os.getcwd()

if platform is 'Windows':
 #path_image = 'C:\\Users\\' + currentUser
 path_image = currentDirectory 
 #path_image = '/user/' + currentUser
 path_image = currentDirectory 
fn = 'image.png'
img = io.imread(os.path.join(path_image, fn))
  • Load MNIST dataset and preprocess images pixels into arrays
# prepare arrays
X_t = []
y_t = []

X_t = numpy.asarray(X_t)
y_t = numpy.asarray(y_t)
y_t = np_utils.to_categorical(y_t, 10)

(X_train, y_train), (X_test, y_test) = mnist.load_data()

# flatten 28*28 images to a 784 vector for each image
num_pixels = X_train.shape[1] * X_train.shape[2]
X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')
X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')
X_t = X_t.reshape(X_t.shape[0], num_pixels).astype('float32')

# normalize inputs from 0-255 to 0-1
X_train = X_train / 255
X_test = X_test / 255
X_t /= 255

print('X_train shape:', X_train.shape)
print ('X_t shape:', X_t.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
print(X_t.shape[0], 'test images')

# one hot encode outputs
y_train = np_utils.to_categorical(y_train)
y_test = np_utils.to_categorical(y_test)

num_classes = y_test.shape[1]
print(y_test.shape[1], 'number of classes')
  • Define helper functions 
# define baseline model
def baseline_model():
 # create model
 model = Sequential()
 model.add(Dense(num_pixels, input_dim=num_pixels, init='normal',activation='relu'))
 model.add(Dense(num_classes, init='normal', activation='softmax'))
 # Compile model
 model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=['accuracy'])
 return model
def build_model(model):
 # build the model
 model = baseline_model()
 # Fit the model
 model.fit(X_train, y_train, validation_data=(X_test, y_test),nb_epoch=10, batch_size=200, verbose=2)
 return model

def save_model(model):
 # serialize model to JSON
 model_json = model.to_json()
 with open("model.json", "w") as json_file:
 # serialize weights to HDF5
 print("Saved model to disk")
def load_model():
 # load json and create model
 json_file = open('model.json', 'r')
 loaded_model_json = json_file.read()
 loaded_model = model_from_json(loaded_model_json)
 # load weights into new model
 if loaded_model:
 print("Loaded model")
 print("Model is not loaded correctly")
 return loaded_model

def print_class(scores):
 for index, score in numpy.ndenumerate(scores):
 number = index[1]
 print (number, "-", score)
 for index, score in numpy.ndenumerate(scores):
 if(score > 0.5):
 number = index[1]
 print ("\nNumber is: %d, probability is: %f" % (number, score))
  • Prepare multilayer perceptron model and compile it
model = baseline_model()
path = os.path.exists("model.json")
  • Check if trained model exists
  • If not train new model, save it and predict image 
if not path:
 model = build_model(model)
 # Final evaluation of the model
 scores = model.predict(X_t)
 print("Probabilities for each class\n")
  • Else load current model and predict image 
 # Final evaluation of the model
 loaded_model = load_model()
 if loaded_model is not None:
 loaded_model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=['accuracy'])
 scores = loaded_model.predict(X_t)
 print("Probabilities for each class\n")

How to Run 

If you downloaded/ cloned the project and you have all prerequisites set up then to run it simply type this command in terminal.

python mnist_mlp_baseline.py

The Prediction Exposed

The predicted output for the image of digit ‘3’ looks like this.

Probabilities for each class

(0, '-', 3.4988901e-07)
(1, '-', 3.7538914e-08)
(2, '-', 0.00072528532)
(3, '-', 0.99788445)
(4, '-', 1.7879113e-08)
(5, '-', 1.3890726e-06)
(6, '-', 2.5650074e-10)
(7, '-', 2.233218e-05)
(8, '-', 0.0012537371)
(9, '-', 0.00011237688)

Number is: 3, probability is: 0.997884


  • If you want to see a 3-D visualization of Multilayer Perceptron Network in action  built with two hidden layers then check this one
  • If you want to see a nice visualization of a shallow/ deep neural network and play with various parameters yourself in a real time then A Neural Network Playground is for you!

 Java Code Geeks