Experimental Evaluation (EKAW 2018 Paper)

Introduction

This page contains information about benchmarking our spatial OQA (Ontology-Mediated Query Answering) system in three Cooperative Intelligent Transport Systems (C-ITS) scenario and related use cases. The C-ITS streams are integrated by an approach called Local Dynamic Map (LDM), which is a standardized integration platform for spatial and stream data.

General Setup

The following components are used to perform the experiments:

Database

The PipelineDB instance is split into a static and a streaming part. The static database includes the ontology definitions and static instances: The stream database uses functionality of PipelineDB's, i.e., Streams and Continuous Views. We modeled the database in a way, that there is a one-to-one mapping from streams → continuous views → TBox concepts/roles: Each stream is made of the columns iid (object identification), x (data point), and tp (time point of the data item). We provide a PipelineDB 9.8.1 database dump that incluses the full DB schema with static instances. Use the PipelineDB loader pipeline -d ldm_2 -f ldm_2.bak to create the database ldm_2. As a prerequisite the database ldm_2 has to be created (e.g., using pgAdmin3) using the parameter Encoding: UTF8 and Template: pipeline.

Ontology (TBox)

Our LDM ontology models in OWL 2 QL an LDM and ITS domain using a layered approach, separating bewtween the concepts (aka classes): ITS features (e.g., intersection topology), geo features (e.g., POIs), geometrical representations (e.g., polygon), actors (e.g., vehicles), events (e.g., accident); and roles: (aka properties) partonomies (e.g., isPartOf), spatial relations (e.g., intersects), connectivity (e.g., connectedTo), and generic roles (e.g., speed).

C-ITS Scenarios

Our experiments are based on the following three scenarios, which were designed in cooperation with domain experts at Siemens Mobility: The scenarios are split into several use-cases that build the base to derive features and requirements. We have derived a set of ten queries that represent the use-cases and take the features and requirements of each case into account.

Scenario Data

Image of 4 intersections scenario For the scenario data, we use a realistic traffic simulation of 4 intersections in a #-shaped layout. The scenario environment was developed with the microscopic traffic simulation PTV VISSIM that allows us to simulate realistic driving behavior and signal phases. The structure of the intersection, driving patterns and signal phases are fixed, but we have adapted the traffic densities by light, medium, and heavy traffic.

We developed an adapter (not published here) that extracts the actual state of each simulation step, allowing us to replay the simulation from the logs. To increase the throughput for the experiments, we are able to replay the simuluation with 5ms, 10ms, 50ms, and 100ms (real-time speed of VISSIM) delay.

For creating the scenario data, we provide a script for the generation of existing vehicles vehicles_gen.py (run once for initialization) and a script for running the simulation generating the streams stream_play_vissim.py. The second script replays a chosen VISSIM simulation log, which we recorded from the simulation. We used low, medium, and high traffic densities for our benchmarks.

Below are the queries (divided by each scenario) that are used for the experiments:

Traffic Statistics

Hazardous Events Detection

ADAS

Decompositions

The hypergraph decompositions for each query are available in the following file. We have named the sub-queries by their types "_N" (normal), "_F" (stream), "_S" (spatial), and "_J" (join). For instance, the decompositions for query q11 is as follows:

Results

We conducted our experiments on a Mac OS X 10.13.4 system with an Intel Core i7 2.9GHz, 8 GB of RAM, and a 500GB SSD. The average of 21 runs for the query rewriting and evaluation time was calculated with no outliers ignored.

Evaluation

The results of our experiments are shown below with the average evaluation time (AET) in seconds for low, medium, and high traffic data replayed with 5, 10, 50, and 100ms delay, where 100ms is the normal replay speed of VISSIM. We show for each conjunctive query the number of atoms and the number of sub-queries including the number of stream sub-queries in brackets.

# Sub
Queries
# Atoms
5
Low
10
Traffic
50

100

5
Medium
10
Traffic
50

100

5
High
10
Traffic
50

100
Logs
q1.1 3 (2) 42 1.35 1.18 0.95 0.86 1.45 1.30 0.99 0.88 1.46 1.35 1.14 0.99 q11
q1.1* 3 (2) 42 1.00 0.95 0.81 0.80 1.03 0.96 0.83 0.80 1.05 1.03 0.87 0.81 q11b
q1.2 6 (2) 43 1.30 1.20 1.01 0.96 1.33 1.24 1.04 1.00 1.41 1.38 1.07 1.01 q12
q1.3 8(2) 44 1.44 1.35 1.15 1.08 1.47 1.37 1.23 1.09 1.45 1.44 1.30 1.20 q13
q2.1 6(2) 43 1.31 1.20 1.01 0.98 1.43 1.29 1.09 0.99 1.48 1.40 1.13 1.02 q21
q2.2 7(2) 45 1.36 1.26 1.05 1.00 1.47 1.29 1.08 1.03 1.51 1.43 1.13 1.06 q22
q2.3 7(3) 50 1.57 1.50 1.27 1.21 1.63 1.53 1.30 1.22 1.72 1.65 1.37 1.27 q23
q2.4 5(2) 46 1.24 1.21 0.98 0.92 1.28 1.24 1.06 0.97 1.28 1.29 1.13 0.99 q24
q2.5 7(3) 43 1.44 1.38 1.16 1.08 1.50 1.41 1.20 1.11 1.55 1.47 1.26 1.17 q25
q3.1 5(2) 43 1.85 1.72 1.40 1.32 1.89 1.79 1.48 1.35 2.06 2.04 1.57 1.38 q31
q3.2 5(3) 63 1.41 1.34 1.23 1.17 1.48 1.43 1.27 1.20 1.56 1.51 1.31 1.21 q32
q3.3 12(5) 43 3.02 2.80 2.42 2.39 3.26 2.98 2.58 2.38 3.36 3.20 2.66 2.44 q33

Each log file contains the details on each run for a particular query, traffic density, and replay delay. For q3.3, we estimate the AET by running q2.1 twice for the the first two rules and add an estimate of 0.4s to evaluate for the third rule, since our approach is geared towards CQ and cannot run a Datalog rules directly.

Two comments regarding the evaluation of the stream atoms:

Sources and Executable

The files for reproducing the experiments are available under shell script, JAR-file, and config.properties.
The sources of our implementation is available under code, where the entry point to the stream QA code is in SemSearchLDM.java.

References

[1] Thomas Eiter, Herbert Füreder, Fritz Kasslatter, Josiane Xavier Parreira, Patrik Schneider. Towards a Semantically Enriched Local Dynamic Map. International Journal of Intelligent Transportation Systems Research. Available online (2018)

[2] Spatial Ontology-Mediated Query Answering over Mobility Streams. In Proc. of ESWC 2017. pp. 219–237 (2017)

[3] Detecting Mobility Patterns using Spatial Query Answering over Streams. Stream Reasoning Workshop at ISWC 2017 (2017)

Contact

Patrik Schneider patrik(at)kr.tuwien.ac.at