Kaitai Web IDE on Windows and Linux

kaitai

A Guide to Kaitai Web IDE

This post is a continuation of my previous post about Kaitai Struct DSL language for description of binary data formats. This post will describe how to download Kaitai Web IDE and run it locally as a web application and  also how to build and run it.

Prerequisites

If you are interested to run Kaitai Web IDE locally or take part in its development then there is a need to install some additional software, such as

  • Anaconda 2/3 that will help install
    • Git
    • Python 2/3
    • Node.js (to be able to build locally)

This guide will explain how to do it on Windows and Linux(Ubuntu). 

In addition the following versions will be used

  • Anaconda with Python 2.7
  • Git 2.14.1
  • Python 2.7
  • Node.js 6.11.0

Anaconda for the rescue

Anaconda is a package manager for various libraries be it Python, Node.js etc. We’ll use it to install all the dependencies we need for Kaitai Web IDE.

1. Download Anaconda from official site

conda.png

For the sake of this tutorial you can chose either Python 2.7 or 3.6 version.

2. Install Anaconda as any application.

Look for newly installed programs and run Anaconda Prompt (Windows)

anaconda.png

3. Open command line or Anaconda prompt and install Git (taken from here)

conda install -c conda-forge git

4. Approve if asked to install any additional packages.

5. Test that Git was installed correctly

C:\Users>git --version
git version 2.14.1.windows.1

6. Clone or download Kaitai Web IDE stable release from GitHub repository.

git clone https://github.com/kaitai-io/ide-kaitai-io.github.io

7. You’ll see output resembling this one

C:\Users>git clone https://g
ithub.com/kaitai-io/ide-kaitai-io.github.io
Cloning into 'ide-kaitai-io.github.io'...
remote: Counting objects: 3003, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 3003 (delta 0), reused 1 (delta 0), pack-reused 2995
Receiving objects: 100% (3003/3003), 5.98 MiB | 5.11 MiB/s, done.
Resolving deltas: 100% (2118/2118), done.

To run Kaitai Web IDE as is 

8. Open the folder where you placed Kaitai Web IDE (in my case

C:\Users\ide-kaitai-io.github.io

)

9. Run following command to lunch web server that will host locally Kaitai Web IDE app

python -mSimpleHTTPServer

10. Go to http://127.0.0.1:8000/

Note: pay attention that instead of port 8000 you may need to use any other such as 8888 if 8000 is used by some application if you see the error below

socket.error: [Errno 10013] An attempt was made to access a socket in a way forb
idden by its access permissions

10 (in case of error). To fix this execute previous command with a port as an argument

python -mSimpleHTTPServer 8888

Go to http://127.0.0.1:8888/

As you can see in the screenshot below I had to run on port 8888 and Kaitai Web IDE is up and running!

8888.png

Build it yourself

If you want to build Kaitai Web IDE yourself then install additional software with Anaconda.

The attempt below failed but the one after it passed. I preserve it for the sake of an argument (it can be done easily on Linux)

11. Install Node.js from here

conda install -c conda-forge nodejs 

12. Approve if asked to install any additional packages.

And there was an error on Windows (why is that?)

CondaError: WindowsError(206, 'The filename or extension is too long')

12 (once again). Ok. Then let’s download and install Node.js from official site.

node.png

13. Open command prompt and test Node.js version

C:\Users\iscamc>node --version
v6.11.3

We are almost on the finish line. To build Kaitai Web IDE

14. Clone or download Kaitai Web IDE stable release from GitHub repository.

git clone --recursive https://github.com/kaitai-io/kaitai_struct_webide

You’ll see output like this

C:\Users>git clone --recursi
ve https://github.com/kaitai-io/kaitai_struct_webide
Cloning into 'kaitai_struct_webide'...
remote: Counting objects: 5175, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 5175 (delta 6), reused 12 (delta 5), pack-reused 5153
Receiving objects: 100% (5175/5175), 9.63 MiB | 4.49 MiB/s, done.
Resolving deltas: 100% (3852/3852), done.
Submodule 'formats' (https://github.com/kaitai-io/kaitai_struct_formats/) regist
ered for path 'formats'
Cloning into 'C:/Users/kaitai_struct_webide/formats'...
remote: Counting objects: 1576, done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 1576 (delta 37), reused 63 (delta 23), pack-reused 1495
Receiving objects: 100% (1576/1576), 431.29 KiB | 2.82 MiB/s, done.
Resolving deltas: 100% (888/888), done.
Submodule path 'formats': checked out 'a3643b677daccfd323f7d9ace998292c9ee51811'

15. Open the folder where you placed Kaitai Web IDE (in my case 

C:\Users\ide-kaitai-io.github.io

)

16. Run following command to install all the JavaScript and TypeScript dependencies with 

npm install

17. Now  compile and run the Web IDE itself

python serve.py --compile

Note: pay attention that instead of port 8000 you may need to use any other such as 8888 if 8000 is used by some application if you see the error below

socket.error: [Errno 10013] An attempt was made to access a socket in a way forb
idden by its access permissions

To fix this error open in any text editor of your choice serve.py file that resides inside Kaitai folder in my case in C:\Users\kaitai_struct_webide and replace PORT=8000 with, say PORT=8888  on line 15.

line_15.png

17 (in case of error). Rerun command below

python serve.py --compile

When everything worked nice you’ll see something like this

C:\Users\kaitai_struct_webide>python serve.py --compile
Starting typescript compiler...
Please use 127.0.0.1:8888 on Windows (using localhost makes 1sec delay)
Press Ctrl+C to exit.
12:57:28 AM - Compilation complete. Watching for file changes.

18. Go to http://127.0.0.1:8888/

Congratulations you’ve build Kaitai Web IDE!

congrats.png

 Java Code Geeks

Web Code Geeks

Advertisements

Reverse Engineering with Kaitai Struct

Reverse engineering the easy way

Imagine you have some kind of 3rd party data storage that you need to understand how to work with and the only thing you have is a detailed description of the protocol using the device. The only problem is that there is no source code available that can make this process easy to accomplish. And what is left is to implement manually this protocol while having lots of trial and error iterations. Next time in similar occasion repeat this difficult process once again. But no worries, there is one tool that comes in handy in situations like this when there is a file or a stream that you want to parse and you want to be able to do it fast.  

Meet Kaitai Struct

First, here comes an official description of Kaitai Struct

Kaitai Struct is a domain-specific language (DSL) that is designed with one particular task in mind: dealing with arbitrary binary formats.

Parsing binary formats is hard, and that’s a reason for that: such formats were designed to be machine-readable, not human-readable. Even when one’s working with a clean, well-documented format, there are multiple pitfalls that await the developer: endianness issues, in-memory structure alignment, variable size structures, conditional fields, repetitions, fields that depend on other fields previously read, etc, etc, to name a few.

Kaitai Struct tries to isolate the developer from all these details and allow to focus on the things that matter: the data structure itself, not particular ways to read or write it.

Features

  • Kaitai is supported on Linux and Windows (not sure about Mac).
  • So far, Kaitai supports generating parsers in following languages
    • C++/STL
    • C#
    • Java
    • JavaScript
    • Perl
    • PHP
    • Python
    • Ruby
  • If you want you are welcome to add one more language to the list

How to use this Kaitai?

In short, to use Katai 

  • You use declarative syntax to describe a data source you want to be able to parse, such as file system or image format or whatever you like, in ksy file. 
  • Then using Kaitai Web IDE or Katai Struct compiler you generate a code in one of the relevant supported languages, such as Java, C#, C++ etc.
  • That’s it. Now use the code to get full access to your data source.

Kaitai REPL (Read–Eval–Print Loop

repl.png

To get a feeling what Kaitai is capable of you can start from playing with Kaitai REPL which has a number of examples showcasing what can be achieved with it, such as parsing doom.wad package files format.

Katai Web IDE

2017_09_17_23_16_07_Kaitai_Web_IDE.jpg

If you think you are ready to start applying Kaitai to real problems then jump into Katai Web IDE which is very nice and easy to use. You can upload there your data source and start writing a description of how the data source is organized. 

This official wiki page will show you the main features or Web IDE.

Kaitai Compiler stand alone 

It is possible to use Katai compiler in a stand alone mode via command line interface of your choice be it on Linux, Windows etc. How to do it is described here.

Resources

mikhail.png

 Java Code Geeks

Banana Classifier with OpenCV on Android

whatsapp-image-2016-11-16-at-13-10-02

Update: BananaRecognition source code at GitHub

Make Banana Recognition Great Again

This post is about going from an idea to its implementation. The idea was to create an application for mobile phone that can detects cars. Now let’s see how it started and where I stand today. I’ll provide a brief description of the steps taken to this very day.

Step 1. Google is Your Friend or Search and You’ll Find It

The journey to banana Haar Feature-based Cascade Classifier started from searching for videos, articles and blogs on how to make object detection and tracking with a mobile phone. In particular Android platform was of interest to me since I participated in development of a few android applications beforehand. Below follow various kinds of resources that were found and how they provided me with a direction for further search.  

Going in a wrong direction

First search results were actually disappointing. There were a bunch of recognition APIs and recognition products for static object recognition  and classification of video for captions generation. To name a few

Those are good for static images but I was looking for something else i.e. dynamic treatment.

YouTube provides hope for feasibility

Looking for a possible implementation for my idea I started to search  further with a little help from Google and found a number of items on youtube and in blogs that were encouraging.

Advanced object detection take one

The video below appeared to be exactly what I imagined in my mind. It is ViNotion object detection from moving vehicle (car detection)

But as I found very fast this company from Netherlands used very capable HW and proprietary SW to accomplish this feat.

So I needed to find something else.

Advanced object detection take two

Next thing I found was this system that was able not only to recognise moving cars but also was able to classify them and provide their speed.

Once again it turned out to be this monstrous HW thing with thermal sensor, laser range finder you name it.

The power of academic research

The search continued and bingo! This was it. What I’ve found was an article with exactly what I wanted. Android phone detecting and tracking cars on the road.

In short they used Android phone powered by Haar-like Feature Detector with additional filters and were able to detect and track cars from the back. But the key elements I learned from this article were

  • It is possible and was done with mobile phone
  • Haar-like Feature Detector with AdaBoost algorithm is a candidate for usage

Step 2. OpenCV and Roman Hošek to the Rescue

It was only a small step to type in Google Search OpenCV on Android to find about the power of Open Source Computer Vision library known as OpenCV to get even closer to my goal. Looking into OpenCV tutorials I understood that it would take me much more time than I wanted to invest to understand how to set it up on Android and how to start development fast. So I continued searching and found a detailed two parts blog post of Roman Hošek describing exactly how to implement Android application for face recognition using OpenCV library.

Not only he described how to implement this application he also provided a link to his GitHub repository  with a working Android application that easily may be imported and build and run in Android Studio.

Step 3. Download. Build. Find Banana Model

I played with Roman’s application and was able to understand exactly what I needed to do to swap face classifier model to another model. Be it bananas or whatever. Bananas are more easy to classify for the newbies. Looking for bananas classifiers was also not so hard. Moreover there are a bunch of blogs in the wild providing a wealth of information on how to scientifically classify bananas and other fruits with Node.js. 

Train your own Cascade Classifier 

As I found Cascade classifier algorithm is pretty universal in the sense that it can recognise not only faces but other objects as you wish. But it means there is a need for custom training.

It happened that Thorsten Ball provided a GitHub repository that described how to train a custom classifier on… bananas. What was so special about this repository was the file banana_classifier.xml which was the last element in solving the puzzle of bananas, sorry car’s recognition.

Know How To Place A Right File Into A Right Place

Having Roman’s app for face recognition and a model for banana recognition in my hands I was able to tweak the Roman’s app to bananas recognition. The comparison of before/ after functionality is provided below.

App with face recognition model

WhatsApp Image 2016-11-12 at 17.40.47.jpeg

App with banana recognition model

WhatsApp Image 2016-11-14 at 22.18.41.jpeg

Step 4. Find a car model or train one yourself. Get rid of bananas

Next step is to train or find a car model and voila the idea is 100% implemented as envisioned.

WhatsApp Image 2016-11-19 at 18.51.05.jpeg

Java Code Geeks

Guess a Digit Game On Android With TensorFlow

Screenshot_2016-11-12-01-02-06_jp.narr.tensorflowmnist.png

Why Machine Learning?

About a week ago I’ve discovered an overwhelming topic of machine learning and since then there’s no stopping for me. This is a wast field that combines mathematics, programming, artificial intelligence, physics you name it. I have to say this is the thing I’ve been looking for so long time. It  has just a right combination of ingredients I’ve described above making it a very attractive, dynamic and interesting field of study and application.

An Idea For A Game

Recently I’ve been involved with applications development for Android and I thought that combining machine learning with Android is a good idea to try. That is why I want to use a basic application developed by César Delgado Fernández that makes use of TensorFlow library on Android and turn it into engaging game for children. Currently César Delgado Fernández app can recognize digits from 0 to 9. Surely, it can be extended to be capable of other things. If you are eager to see it in action you may download and import it into Android Studio then build. It worked fine for me. If you have any difficulty let me know. So stay tuned for a brand new game from neaapps in following weeks.

Mobile Classifier Application Demo From Google

Discover more about Mobile TensorFlow here. By the way if you want to play with TensorFlow on Android you may try to follow the guide provided here. This application use mobile phone camera to classify objects and then provides captions with probabilities for recognized objects.

Hot Updates In Machine Learning

E-learning for Free

E-Books for Free

  • Deep Learning  is a book by Ian Goodfellow and Yoshua Bengio and Aaron Courville
    the people who are foremost experts in the field of machine learning.

What’s next?

Next post will be about setting and using SciPy stack on Windows and much more.

Machine Learning For All

The Rise of the Machines

In recent decade we are witnessing the resurrection of interest in neural networks due to advances in computing hardware such as GPU accelerators and availability of large data sets such as Facebook, YouTube etc. for training those networks. As a result there are a great number of research articles and practical applications emerged in the filed of artificial intelligence that found fruitful usage at Google, Facebook, Amazon etc.

Meet TensorFlow Library

As a forerunner in the filed of applied machine learning Google has developed and open sourced TensorFlow software machine learning library that is accessible for all to tackle the problems in computer vision, natural speech processing and more. If you find this information interesting then following steps will help you to start playing with this library. The library is written in C++ with APIs available in C++ and Python. Actually most  education resources that are available online for TensorFlow use Python.

Installation Is Fast

Tensor Flow is developed with Linux and OS X in mind and struggles on Windows. That is why the steps below taken from official manual were run on Ubuntu 15.10.

  • Open terminal in Ubuntu/ Linux 64 bit and install pip package management system for Python
$ sudo apt-get install python-pip python-dev
  • Since I had no GPU on my  64 bit machine I chose this binary
# Ubuntu/Linux 64-bit, CPU only, Python 2.7
$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0rc2-cp27-none-linux_x86_64.whl
  • Now let’s install TensorFlow library. Be prepared it will take a while
# Python 2
$ sudo pip install --upgrade $TF_BINARY_URL
  • If installation went flawlessly which it was in my case it is possible to test it typing
$ python
  • When you’ll see ‘>>>’ it means Python interpreter is running and you can play with it
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow! I've made it so far.')
>>> sess = tf.Session()
>>> print(sess.run(hello))
Hello, TensorFlow! I've made it so far.
>>> a = tf.constant(10)
>>> b = tf.constant(32)
>>> print(sess.run(a + b))
42

Hot update: TensorFlow 0.12 available on Windows 7/10

Today we announced native Windows support in TensorFlow 0.12, with packages for Python 3.5. If you have installed the 64-bit version of Python 3.5 (either from Python.org or Anaconda), you can install TensorFlow with a single command:

C:\> pip install tensorflow

How to Use It?

If you want to get a feeling of what TensorFlow is capable of first hand then try to follow the Image Recognition official tutorial or do the steps below to recognize your most favorite image (do not try it with this one). TensorFlow Inception v-3 model will try to recognize the image provided by you and emit the caption for the image from the highest to lowest probability. For example, if you provide an image of Italian Alps then the result will be something along these lines.  

background

alp (score = 0.73395)
valley, vale (score = 0.18973)
cliff, drop, drop-off (score = 0.00309)
promontory, headland, head, foreland (score = 0.00171)
lakeside, lakeshore (score = 0.00154)

Try It yourself

  • The models and examples generally installed at this path
/usr/local/lib/python2.7/dist-packages/tensorflow
/usr/local/lib/python2.7/site-packages/tensorflow
  •  To figure out the exact location use this directive for python 2.7 (change ‘python’ to ‘python3’ for Python 3)
$ python -c 'import os; import inspect; import tensorflow; print(os.path.dirname(inspect.getfile(tensorflow)))'
  •  For Image Recognition you need to run classify_image.py script which can be found at this path
/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/imagenet
  • Place the image in JPEG format you want to be recognized anywhere you like. In my case it was at this destination
/home/me/Pictures/image.jpg
  • Then run Inception v-3 model on that image this way. Pay attention that ‘– -image_file’ argument is there to indicate the path to the image
$ python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/imagenet/classify_image.py --image_file /home/me/Pictures/image.jpg

Teach Yourself Machine Learning In Ten Years

So you like what you did with TensorFlow and you want to learn the subject in more depth. Congratulations, since there are more resources than you can digest in your lifetime. What I personally found useful for me so far are.

1. Start with the TensorFlow site itself where you can find tutorials, guides and APIs description.

2. If you need a more thorough introduction to Machine Learning try to get helped by Udacity free courses on the subject

3. If you are like me and likes books and rigorous theoretical background then this book by Michael Nielsen is just for you.

4. Machine Learning Mastery site by Jason Brownlee is definitely a place to visit

5. If you are an expert in the filed of Machine Learning then maybe you’ll find this peculiar blog by Chris Olah from Google Brain Team useful.

More Posts Are Expected 

Stay in touch.

Java Code Geeks

Get Ready For Machine Learning

Welcome To The Machine

In the upcoming weeks I’ll plan to post materials about Machine Learning be it deep or shallow etc.

A Few Bits Here And There

  • Just to warm your interest start by skimming over this insightful blog by Chris Olah a member at Google Brain Team.
  • Then take a look at TensorFlow open source machine learning library developed in Google
  • Do not forget to check a very different and yet somewhat similar to Neural Networks  Numenta’s Hierarchical Temporal Memory (HTM) model
  • Those of you who want to know more about Numenta 

More Stuff is Coming

So stay tuned.

My Zoo has been just published

jungle12.png

Update to the latest version 1.3 at Google Play

What’s this all about?

My Zoo application will help your kids to learn about various animals, their names and sounds and it is out there at Google Play. This game was brought to you by raising star application company neaapps

Features

  • Animals names are in English, Russian and Hebrew
  • Dozens of jungle and domestic animals with their sounds
  • Interactive animation that will captivate your kids
  • Supported on tablets
  • No ads
  • Addctive and fun

What’s new in this version

To make navigation super fast and easy for kids two buttons were added to farm and jungle screens.

To jump to farm animals from jungle tap on the sheep’s icon

to_farm.png

To jump to jungle animals from farm animals tap on the parrot’s icon

to_jungle.png

Take a ride

Do try this at home or anywhere at all. Beware that the sounds may cause people around you to start smiling and laughing.