Data Visualization with the ELK stack and Unity

Elk stack pictureNot many people use Unity(or really any game engine) for data visualization. I think that the main reason for this is that Unity is not marketed as a data visualization tool, and there is already a large market for such things that companies would rather use. I am currently exploring the different possibilities for interactive data visualization using Unity and the Logstash pipeline. Logstash is just one part of something much bigger called the ELK stack(Elasticsearch, Logastah, and Kibana).

For more information on what exactly the ELK stack is, or want a great guide to installing it, check out my friend’s website for help. This was a HUGE resource for me when it came to installing the ELK stack on CentOS (which I have never used before 3 days ago) .

What makes Logstash so great in my eyes, is that it outputs in JSON format that is easily accessible. You can get JSON data from the ELK stack by posting an HTTP request in Unity via the WWW component.  You can also do this more broadly in C# using the HttpWebRequest class. The reason I use WWW is because it is more optimized for game engines. After you store this data in a string (depending on what post query you use), you can use Unity’s JsonUtility to parse through it and make it accessible like any other C# data. I made a more detailed post about how to this here.

Once you have the data in C#, the possibilities are endless. A particularly interesting application of this is what I am doing on my current co-op, which is network data visualization. Reading the logs from tools like Bro, Snort, and Tshark into Logstash via Filebeat, I can effectively visualization the devices that are on the network, and the netflow data going between them. Since we are using Unity, we can be as creative as we want with how we show this, instead of just a 2D graph like most network visualizations.

I am particularly interested in using VR to show the network data. Image walking through some netflow data in real time, that would be pretty cool!

JSON data in Unity

Json logo

The project that I am currently working on calls for being able to parse JSON data in Unity. Now, you may be thinking, “There is an actual documentation page on this”, and you are not wrong. In fact, it’s right here.

However, if you are new to JSON and the utility in Unity, then it can be a bit confusing to set up the C# code with more complex JSON data.

example output
Example output from my Log Files

As you can see, there are a couple ‘children’ in this JSON data like “_source” and “beat”, and also an array called “hits”.  To set this up in C# is very simple, but it is lacking the actual Unity Documentation.

[System.Serializable]
public class Json_Data
{
 public int took;
 public bool timed_out;
 public HitsParent hits; 
}

[System.Serializable]
public class HitsParent
{
 public int total;
 public HitsData[] hits;
}

[System.Serializable]
public class HitsData
{
 public string _index;
 public string _type;
 public string _id;
 public Source _source;
}

Make sure that each variable’s name in C# matches with the JSON field name, or it will actually mark all the variables that come after that as null.

Take note that it is bad practice to have multiple classes within the same file, but for JSON it is a bit cleaner to put it all in one class if you will have multiple data types.

Every class needs to be serialize, because that is how Unity parses the data. To learn more about why each class needs to be Serializable, go to the Unity documentation.