Reverse Engineering with Kaitai Struct

Reverse engineering the easy way

Imagine you have some kind of 3rd party data storage that you need to understand how to work with and the only thing you have is a detailed description of the protocol using the device. The only problem is that there is no source code available that can make this process easy to accomplish. And what is left is to implement manually this protocol while having lots of trial and error iterations. Next time in similar occasion repeat this difficult process once again. But no worries, there is one tool that comes in handy in situations like this when there is a file or a stream that you want to parse and you want to be able to do it fast.  

Meet Kaitai Struct

First, here comes an official description of Kaitai Struct

Kaitai Struct is a domain-specific language (DSL) that is designed with one particular task in mind: dealing with arbitrary binary formats.

Parsing binary formats is hard, and that’s a reason for that: such formats were designed to be machine-readable, not human-readable. Even when one’s working with a clean, well-documented format, there are multiple pitfalls that await the developer: endianness issues, in-memory structure alignment, variable size structures, conditional fields, repetitions, fields that depend on other fields previously read, etc, etc, to name a few.

Kaitai Struct tries to isolate the developer from all these details and allow to focus on the things that matter: the data structure itself, not particular ways to read or write it.

Features

  • Kaitai is supported on Linux and Windows (not sure about Mac).
  • So far, Kaitai supports generating parsers in following languages
    • C++/STL
    • C#
    • Java
    • JavaScript
    • Perl
    • PHP
    • Python
    • Ruby
  • If you want you are welcome to add one more language to the list

How to use this Kaitai?

In short, to use Katai 

  • You use declarative syntax to describe a data source you want to be able to parse, such as file system or image format or whatever you like, in ksy file. 
  • Then using Kaitai Web IDE or Katai Struct compiler you generate a code in one of the relevant supported languages, such as Java, C#, C++ etc.
  • That’s it. Now use the code to get full access to your data source.

Kaitai REPL (Read–Eval–Print Loop

repl.png

To get a feeling what Kaitai is capable of you can start from playing with Kaitai REPL which has a number of examples showcasing what can be achieved with it, such as parsing doom.wad package files format.

Katai Web IDE

2017_09_17_23_16_07_Kaitai_Web_IDE.jpg

If you think you are ready to start applying Kaitai to real problems then jump into Katai Web IDE which is very nice and easy to use. You can upload there your data source and start writing a description of how the data source is organized. 

This official wiki page will show you the main features or Web IDE.

Kaitai Compiler stand alone 

It is possible to use Katai compiler in a stand alone mode via command line interface of your choice be it on Linux, Windows etc. How to do it is described here.

Resources

mikhail.png

Advertisements

A Digest of Deep Learning Pearls

All you need is time and GPU

Try to allocate time for these thought provoking Deep Learning papers. Part of them with try it yourself implementation at GitHub.

1. Try it yourself at home or anywhere at all (with GPU)

Transformer more than meet the eye!
– A novel approach to language understanding from Google Brain(via David Ha)
It is a very interesting solution for an old linguistic/ syntactic challenge (anaphora) with Deep Learning. More detailed explanation of anaphora resolution.
– Based on “Attention is all you needpaper

2. Learning To Remember Rare Events

An interesting approach to introduce memory module into various types of Deep Learning architectures to provide them with life long learning.

3. One Model To Learn Them All

A unified Deep Learning model that is capable of being applied to inputs from various modalities. It is a one step closer toward general DL architectures.

4. Meet Fashion-MNIST

Finally, it is time to ditch MNIST in favor of Fashion-MNIST

Which is better from a number of aspects. Which one? Find yourself.

**Note:

If you haven’t noticed the one thing in common to all of these items except for one is
Łukasz Kaiser researcher from Google Brain.

 Java Code Geeks

NLP is Natural Language Processing

Get ready for a real NLP

I am back to blogging and have a motivation to post a number of posts (or at least one) on the subject of Natural Language Processing. Upcoming posts also will contain information on recurrent neural networks such as LSTM. So stay tuned.

For now, check this out

If you are into Natural Language Processing (NLP) then you may find links below useful.

Papers

1. Attention Is All You Need paper in arxiv.

 

 

Deep Le… Space, Do You Copy?

600px-AS17-Flag_shots

There are other things too

In the middle of Deep Learning rush we forget that there are other things on this planet and off it that are fascinating. That’s right, I want to share with you the best materials I saw so far on Moon exploration that are highly recommended.

Books From Apollo Participants 

There are quite a few books written about US space program. But there are few that are really good. I’ve chanced to read some of them and below follow the best ones in my opinion.

The Last Man On The Moon Book

last1

The Last Man on the Moon: Astronaut Eugene Cernan and America’s Race in Space

This book is very special and it is a memoir by Gene Cernan the commander of Apollo 17. He was literally the last person to walk on the moon.

ProsThere is a special atmosphere in this book. The descriptions are so vivid and colorful. Gene Cernan was deeply touched by lunar visits since he was there twice on Apollo 10 and then Apollo 17. It is available on Kindle.

                                               Cons. It finished so fast. (No photos in the book)

The Last Man On The Moon Movie

lastmanmoon.jpg

There is also a movie named the same which may be found for free on the internet or bought here. Here is the trailer.

Two Sides of the Moon: Our Story of the Cold War Space Race

2side

Two Sides of the Moon: Our Story of the Cold War Space Race

This book combines recollections by Apollo 15 commander David Scott and his contemporary Alexi Leonov who was the first man to walk in space.

Pros. Very interesting book because of complementing accounts provided by both distinguished persons. Available in Kindle format.

Cons. Not a single photo.

From The Other Side

failureFailure Is Not an Option: Mission Control From Mercury to Apollo 13 and Beyond

The book below provides very different account of the matters described in the books above. It is written by Gene Kranz the Flight throughout entire US space program starting from Mercury and ending in Shuttle era.

Pros. The more technical book than astronauts accounts. Available on Kindle.

Cons. No photos again.

Documentaries that cannot be missed

  1. EARTHRISE: The First Lunar Voyage – documentary about Apollo 10.
  2. Apollo 13 Documentary 1958 – as it was portrayed by NASA.
  3. Apollo 15 Remembered 40 Years Later – documentary for Apollo 15 featuring Neil Armstrong and others.
  4. In The Shadow of The Moon – british documentary with interesting stuff.
  5. The Last Man On The Moon – documentary featuring Gene Cernan.
  6. Failure Is Not An Option A Flight Control History of NASA – documentary featuring flight controllers and Gene Kranz.
  7. Moon Machines – tools that made lunar program possible and people behind them.
  8. From The Earth To The Moon – a series produced by Tom Hanks. 

Not because they are easy, but because they are hard!

This post can’t be finished without the full inspirational to say the least speech by John F. Kennedy. It is incomparable to the current president of the US. It is a giant  speech for a president and a giant gap between then and now. 

potus.png

President Kennedy’s Speech at Rice University

No winter but AI global warming

formula

Name things for what they are

Is Deep Learning rage simply a bubble or is this time it here for a long time to stay. As researchers proposed first let’s change the Deep Learning title into the more humble and exact Multilayered Network for Functions Approximation. Now it sounds more practical and there is no sign of hype. Then check to what fields those networks were applied and see if it is diverse and if the algorithms used are universally applicable. Check the number of articles published that have a real essence within them. If you’ve got ‘yes’ as an answer to those questions then it feels like finally those approaches are really usefull.

What’s next?

This post will be updated in a near future. Meanwhile check the posts by Carlos E. Perez from IntuitionMachine.com that writes extensively on the subject and do not forget to check his ‘The Deep Learning Playbook

Wind of Deep Change

Welcome to the world of Machine and Deep Learning

Following my transition to another continent in near future I’ll be able to focus more on Machine and Deep Learning being a technical editor at renowned Machine Learning Mastery site authored by Dr. Jason Brownlee. It means you can expect more posts on machine learning to come especially on LSTM and recurrent neural networks.

What is it like to be a technical editor?

Throughout my career I’ve been a SW test engineer and SW developer but in parallel I’ve been busy helping to edit books as Jumping Into C++ by Alex Allain and other projects, such as Kindle Optimizer Chrome extension. So becoming a technical editor in machine learning field is just a logical step to make. Actually technical editor is a bit like a QA engineer and a developer at once since you have to understand how Python code is working to make that LSTM to be able to predict time series values and to be a test engineer to make the content and the code to be as good as it can be. In addition, there is a kind of freedom that regular tester or developer do not possess which is to suggest changes to the author which may be meaningful and influential. Most importantly, technical editor deals with the raw content of a future article, a blog post or a chapter from the book that millions of people may read and it provides you with the understanding of the responsibility that you bear on your shoulders. The corrections that you make may influence readers and make their experience pleasant or not.

Why machine or deep learning after all?

Technical editing as testing or programming is a universal position since it can be successfully applied to various topics in those fields, but machine learning has the proper ingredients of math, programming and future potential that makes it very attractive.

Stay tuned as John Sonmez says

So if you follow this blog stay around the corner to be up to date with the current progress in Deep Learning field and if you care check this public Deep Learning for All group at Facebook where I share latest and in my view greatest news coming from Deep Learning fruitful field.

 

 

How to achieve a goal?

Set a goal

Set any  goal that does not contradict known laws of physics, though remember that not all laws are known to us. 

Create a plan

Write a quick plan for a goal. Detailed or not it doesn’t matter since it will be refined in time.

Remember this while acting on a plan

A goal will be achieved by a plan while moving towards it

  • Gradually
  • Consistently
  • Constantly

It is a great force

Acting in this way is like being a force of nature.