About this tutorial
What will you learn ?
With this tutorial, you'll be able to:
- start an ARLAS-Exploration stack
- Index some AIS data in Elasticsearch
- Reference the indexed AIS data in ARLAS
- Create a view of ARLAS-wui (a dashboard) to explore the AIS data using ARLAS-wui-hub and ARLAS-wui-builder
What will you need ?
You will need :
- docker & docker-compose
What will you get ?
Exploration app created in this tutorial
Let's explore some boats position data, provided by Danish Maritime Authority.
This tutorial is based on AIS data emitted from 11/20/2019 to 11/27/2019. We extracted boats positions having the following MMSI :
First, download all the data you need on the Danish Maritime Authority FTP :
# Example for the 11/20/2019 curl -O ftp://ftp.ais.dk/ais_data/aisdk_20191120.csv
And extract only desired MMSI :
cat aisdk_20191120.csv | grep -E "257653000|265177000|220051000|240305000" > ais_data.csv
If you don't want to create your custom extract you can use the one we provide. ( view Setup section).
We built a subset named
ais_data.csv. It contains around 162192 boats positions described with 26 columns.
Example of some columns:
- Timestamp: Moment when the position is emitted
- MMSI: Identifier of the boats emitter
- Name: Name of the boat
- Ship type: Type of the boat
A line of the csv file looks like:
|Timestamp||Type of mobile||MMSI||Latitude||Longitude||Navigational status||ROT||SOG||COG||Heading||IMO||Callsign||Name||Ship type||Cargo type||Width||Length||Type of position fixing device||Draught||Destination||ETA||Data source type||A||B||C||D|
|20/11/2019 06:45:09||Class A||240305000||55.931783||17.345067||Under way using engine||0.0||10.5||257.0||259||9288710||SYEF||DELTA CAPTAIN||Tanker||""||44||249||GPS||10.0||FOR ORDERS||22/11/2019 06:00:00||AIS||216||33||22||22|
Exploring AIS data
We will explore this data using ARLAS.
- Create a repository dedicated to this tutorial
mkdir ARLAS-stack-ais-tutorial cd ARLAS-stack-ais-tutorial
- If you didn't download and extract data from ftp server, you can download this csv file we provide :
curl -O -L https://github.com/gisaia/ARLAS-stack-ais-tutorial/raw/develop/data/ais_data.csv
- Check that
ais_data.csvfile is downloaded
ls -l ais_data.csv
- Download last version of Exploration stack ans unzip it
(curl -O -L https://github.com/gisaia/ARLAS-Exploration-stack/archive/develop.zip; unzip develop.zip)
- Check that the stack is downloaded
ls -l ARLAS-Exploration-stack-develop
1. Starting ARLAS Exploration Stack
- Start the ARLAS stack
2. Indexing AIS data in Elasticsearch
ais_indexindex in Elasticsearch with
curl https://raw.githubusercontent.com/gisaia/ARLAS-stack-ais-tutorial/develop/configs/ais.es_mapping.json | curl -XPUT http://localhost:9200/ais_index/?pretty \ -d @- \ -H 'Content-Type: application/json'
You can check that the index is successfuly created by running the following command
curl -XGET http://localhost:9200/ais_index/_mapping?pretty
- Index data in
ais_data.csvin Elasticsearch. For that, we need Logstash as a data processing pipeline that ingests data in Elasticsearch. So we will download it and untar it:
( curl -O https://artifacts.elastic.co/downloads/logstash/logstash-7.4.2.tar.gz ; tar -xzf logstash-7.4.2.tar.gz )
- Logstash needs a configuration file (
ais2es.logstash.conf) that indicates how to transform data from the CSV file and index it in Elasticsearch.
curl https://raw.githubusercontent.com/gisaia/ARLAS-stack-ais-tutorial/develop/configs/ais2es.logstash.conf -o ais2es.logstash.conf
- Now we can index the data:
cat ais_data.csv \ | ./logstash-7.4.2/bin/logstash -f ais2es.logstash.conf
- Check if 162189 AIS positions are indexed:
curl -XGET http://localhost:9200/ais_index/_count?pretty
ais_index in ARLAS
ARLAS-server interfaces with data indexed in Elasticsearch via a collection reference.
The collection references an identifier, a timestamp, and geographical fields which allows ARLAS-server to perform a spatial-temporal data analysis
- Create a AIS collection in ARLAS
curl "https://raw.githubusercontent.com/gisaia/ARLAS-stack-ais-tutorial/develop/ais_collection.json" | curl -X PUT \ --header 'Content-Type: application/json;charset=utf-8' \ --header 'Accept: application/json' \ "http://localhost:81/server/collections/ais_collection?pretty=true" \ --data @-
- Check that the collection is created using the ARLAS-server
curl -X GET "http://localhost:81/server/collections/ais_collection?pretty=true"
4. Create a dashbord to explore
AIS data with ARLAS
ARLAS stack is up and running and we have ais position data available for exploration. We can now create our first dashboard composed of - a map to observe the boats positions' geographical distribution - a timeline presenting the number of boats positions over time - a search bar to look for boats by their names for instance - some widgets to analyse the data from another axis such as the speed distribution.
To do so, let's go to ARLAS-wui-hub and create a new dashboard named
figure 0: Creation of a dashboard in ARLAS-wui-hub
After clicking on Create, you are automatically redirected to ARLAS-wui-builder to start configuring your dashboard.
Choosing the collection
The first thing we need to do is to tell ARLAS which collection of data we want to use to create our dashboard
figure 1: Choose collection
in our case we choose the
As a first step, I'll set the map at zoom level 13 and the map's center coordinates at Latitude=57,451545 and Longitude=10,787131. This way, when loading my dashboard in ARLAS-wui, the map will be positionned over Danmark.
figure 2: Map initialisation
For now, the map is empty. The first thing we want to find out is where the boats are ?
figure 3: Layer view
To do so, let's add a layer named
Boats to visualise the boats positions.
In the Geometry section, choose the
location features geo-field
figure 4: Adding a Gemetric features layer named 'Boats'
Now, let's define the layer's style. As a starter, we choose the best representation of our geometries: Boats positions are points. We also choose a fixed color (green for instance) and a fixed radius of 5 pixels
figure 5: Customising 'Boats' style
After clicking on Validate, our first layer is created
figure 6: New layer 'Boats' is created
We can go and preview the layer in Preview tab
figure 7: Preview of 'Boats' layer
We see now where the boats are passing by thanks to this layer
Let's find out the time period when these positions were emitted.
For that, let's define a timeline: a histogram that will represent the number of boats positions over time.
For the x-Axis we choose the timestamp field and for the y-Axis we choose Hits count: the number of positions in each bucket. We set 50 buckets in this example
figure 8: Define timeline
In the Render tab we can set a title for the timeline, date format and the histogram type.
figure 9: Rendering of timeline
Search Bar configuration
To define the search bar we can set :
- the placeholder string;
- the field used to seach keywords
- the field used to autocomplete the searched words
figure 10: Define search bar
Save the dashbord and start exploring in ARLAS-wui
Now we defined :
- 'Boats' layer in the map
- the timeline
- the search bar
Let's save this dashboard by clicking on the 'Disk' icon at the left-bottom of the page.
If we go back to ARLAS-wui-hub, we'll find the Boats dashboard created.
figure 11: List of created dashboards
We can now View it in ARLAS-wui
figure 12: Visualize in ARLAS-wui
Going further with the map
We now see the boats positions on the map but we can't distinguish the boats! Let's change that by setting the color from the ship type.
Let's go back to the dashboard builder and edit 'Boats' layer
figure 13: Edit a layer
In the Style section, we choose Generated colors option that will automatically generates a hex color from the chosen field values. For our case, we choose
Ship type field
figure 14: Setting the positions colors by ship type
After saving the layer, we can preview it again and see that now we have three ship types :
- Cargo (blue)
- Tanker (purple)
- Passenger (yellow)
figure 15: Preview generated ship types colors
On the other hand, it could be interesting to analyse the boats speed during their trajectories.
To answer this question, let's create a dedicated layer!
Before doing that, we will first create a Visualisation set, a space where to organise layers that have to be displayed/hidden together. For instance, the 'Boats' layer is put under the 'All layers' visualisation set
figure 16: List of visualisation sets
For our speed layer, let's create a new visualisation set called 'Speed'
figure 17: Adding a new visualisation set
Now let's create our new layer that will allow us to analyse the boats positions speeds.
figure 18: Adding a new layer for boats positions speed
Style section, we interpolate the position colors with interpolation field values. In our case we interpolate the color with the
SOG (speed over ground) field.
We normalize the SOG values and we choose a color palette
figure 19: Interpolating positions colors to positions speed
After saving this layer we can preview it in the Preview tab and start our analyse!
figure 20: Preview of boats positions according to the speed
Geo Big Data
For this tutorial, we only have ~160 000 boats positions to explore. This allowed us to display the boats positions directly on the map.
But what to do in case we had millions of positions to display ?
It would be very difficult to display them all as it would be very heavy to request all that data at once and the browser will not be able to render as many features. We will end up loosing the user experience fluidity.
Most importantly, loading millions of boats positions on the map will not be necessarily understandable: we cannot derive clear and synthesized information from it.
That's why ARLAS proposes a geo-analytic view: we can aggregate the boats positions to a geographical grid and obtain a geographical distribution !
Let's create a dedicated layer for boats positions geographical distribution.
figure 21: Creating a geographical distribution layer
We choose to aggregate
location geo-field to a geographical grid and we choose a fine granularity for this grid.
We will display on the map the grid's cells.
Let's define the style of these cells in
figure 22: Creating a geographical distribution layer
We interpolate the cells colors to the number of boats positions in each cell. That's why we choose Hits count that we normalise and choose a color palette
After saving this layer, we can again visualise it and explore where the positions are geographically
figure 23: Boats positions geographical distribution
We focused on the geographical and temporal analysis. We can also explore other dimensions of the data.
Let's see what does the heading distribution of these positions looks like.
To do so we need to create a histogram. ARLAS proposes to organise all the histograms and other widgets in an analytics board.
We can split the analytics board into tabs. Let's create a tab called 'Tracking' where will add our Heading distribution histogram
figure 24: Creating tab in Analytics board
Once the tab is created, we can add in it a group of widgets. Let's name it 'Heading'
figure 25: Creating a group in Analytics board tab
Let's now create our histogram
figure 26: Choosing a histogram for heading distribution
We can give a title to the Heading distribution histogram
For the x-Axis we choose
Heading field and for the y-Axis we choose
Hits count: the number of positions in each bucket.
figure 27: Defining heading distribution histogram
When we save the histogram we automatically get a preview of it in the analytics board!
figure 28: Preview heading distribution histogram
We can now save the dashbord again using the 'Disk' icon at the left-bottom of the page and view it ARLAS-wui
figure 29: Exploring Boats dashboard in ARLAS-wui
As you can see we created a simple dashboard to start exploring AIS data!
Check out a more sophisticated dashboard about the AIS data demo space!
You can get inspired from our different demos to build other map layers and other widgets.