Experimental Evaluation (ESWC 2017 paper)


This page contains information about benchmarking our spatial OQA (Ontology-Mediated Query Answering) system for mobility streams. Mobility streams are integrated by an approach called Local Dynamic Map (LDM), which is a standardized integration platform for spatial and stream data in Cooperative Intelligent Transport Systems (ITS) .

General Setup

The following components are used to perform the experiments:


The PipelineDB instance is split into three separate parts:

Ontology (TBox)

Our LDM ontology models in OWL 2 QL an LDM and ITS domain using a layered approach, separating bewtween the concepts (aka classes): ITS features (e.g., intersection topology), geo features (e.g., POIs), geometrical representations (e.g., polygon), actors (e.g., vehicles), events (e.g., accident); and roles: (aka properties) partonomies (e.g., isPartOf), spatial relations (e.g., intersects), connectivity (e.g., connectedTo), and generic roles (e.g., speed).


Our experiment is based on two scenarios of monitoring vehicles and traffic lights (a) on a single intersection and (b) on a network of locally connected intersections, both managed by a single roadside C-ITS station.

Scenario (a)

We have a T-shaped intersection that represents a real-world deployment of C-ITS stations in Vienna. It connects two roads with 13 lanes and 3 signal groups that are linked to the lanes. We developed a synthetic data generator that simulates the movement of 10, 100, 500, 1000, 2500, and 5000 vehicles on a single intersection updating the streams averagely 50ms. This allows us to generate streams with up to 10000 data points per sec. and stream. We chose random starting points and simulated linear movements on a constant pace, creating a stream of vehicle positions. We also simulated simple signal phases for each traffic light that toggle between red and green every 3 secs. The aim of this scenario is to show for simple driving patterns the scalability of our approach in the number of vehicles.

We provide two python scripts for generating the data, vehicles_gen.py generates the vehicles (including their brand) as instances, and stream_gen_syn.py generates the vehicle movements and signal phases. The movement scripts can be run in parallel (using --lower --upper to define vehicle id ranges) to generate a large amount of stream data, since a single generator instance can not create large throughput.

Scenario (b)

In (b), we use a realistic traffic simulation of 9 intersections in a grid, developed with the microscopic traffic simulation PTV VISSIM that allows us to simulate realistic driving behavior and signal phases. The intersection structure, driving patterns and signal phases are more complex, but the number of vehicles is lower (max. 300) than in (a), as we quickly have traffic jams. We developed an adapter to extract the actual state of each simulation step, allowing us to replay the simulation from the logs. To vary data throughput, we ran the replay with 0ms, 100ms (real-time), 250ms and 500ms delay. For illustration, we provide a recording of 12s in the simulation.

For this scenario, the vehicle generation script is the same as in (a), and the VISSIM simulation can be generated using stream_gen_vissim.py using the logs of the 9-intersection simulation.


The following queries represent the input for the experiments and are described below:


The following list shows the hypergraph decomposition for each query:


We conducted our experiments on a Mac OS X 10.6.8 system with an Intel Core i7 2.66GHz, 8 GB of RAM, and a 500GB HDD. The average of 11 runs for the query rewriting and evaluation time was calculated with the largest outlier ignored.

Evaluation (a)

The results shown are for scalability scenario (a) and present query type, size of rewritten atoms, the number of subqueries, and the average evaluation time (AET) in seconds for n vehicles. Note, that all the queries with multiple stream atoms (q4, q5, q6, and q9) have a builtin delay of 0.2s since the Continuous Views of PIPELINEDB do return less or no result, if queried immediately after each other.
Each conjunctive query has sub-queries of different types, where O, S, resp. F represents ontology, spatial, resp. stream sub-queries. Marked with (*) are the queries that only contain "slow" streams generated by traffic lights. Also attached are the logs of the executions, where the details on each evaluation step is documented.

Type # Sub
# Atoms
n Veh.


q1 O,F 1 1 0.85 0.82 0.91 1.05 1.22 1.58 a_q1
q2 O,F (*) 1 6 0.83 0.83 0.83 0.83 0.83 0.83 a_q2
q3 O,S,F 3 23 0.89 0.87 1.00 1.25 1.39 1.74 a_q3
q4 O,S,F 3 22 1.10 1.09 1.24 1.53 1.81 2.32 a_q4
q5 O,S,F 3 42 1.11 1.10 1.26 1.39 1.90 1.92 a_q5
q6 O,S,F 7 52 1.39 1.39 1.49 1.69 2.36 2.28 a_q6
q7 O,F (*) 6 69 1.16 1.16 1.16 1.16 1.16 1.16 a_q7
q8 O,S 9 73 0.92 0.94 1.30 1.43 1.72 2.19 a_q8
q9 O,F 9 105 1.67 1.73 1.99 2.06 2.49 2.97 a_q9

A summary of all the logs for (a) is given in the spreadsheet, where we also provide details on the calculation of the AET.

Evaluation (b)

The results shown are for VISSIM scenario (b) with the average evaluation time delay in ms in the 5,6,7, and 8 column.

Type # Sub
# Atoms
ms delay
q1 O,F 1 1 0.78 0.74 0.73 0.71 b_q1
q2 O,F (*) 1 6 0.77 0.77 0.72 0.71 b_q2
q3 O,S,F 3 23 0.83 0.81 0.77 0.75 b_q3
q4 O,S,F 3 22 1.02 1.00 0.95 0.93 b_q4
q5 O,S,F 3 42 1.05 1.00 0.98 0.96 b_q5
q6 O,S,F 7 52 1.40 1.28 1.26 1.25 b_q6
q7 O,F (*) 6 69 1.15 1.12 1.11 1.09 b_q7
q8 O,S 9 73 0.99 0.98 0.92 0.91 b_q8
q9 O,F 9 105 1.71 1.68 1.66 1.63 b_q9

A summary of all the logs for (b) is given in the spreadsheet

Sources and Executable

The files for reproducing the experiments are available under shell script, JAR-file, and config.properties.
The sources of our implementation is available under code, where the entry point to the stream QA code is in SemSearchLDM.java. We will also provide later a full installation environment using gradle.


[1] Thomas Eiter, Herbert Füreder, Fritz Kasslatter, Josiane Xavier Parreira, Patrik Schneider. Towards a Semantically Enriched Local Dynamic Map. In Proc. of 23rd ITS World Congress (ITSWC 2016).

[2] Thomas Eiter, Thomas Krennwallner, Patrik Schneider. Lightweight spatial conjunctive query answering using keywords. In Proc. of ESWC 2013


Patrik Schneider patrik(a)kr.tuwien.ac.at