Benchmark Results (June 2017)

We compare Alpha (git-commit: 037b3f9), Omiga (git-commit: a65421f), ASPeRiX (2.5), and Clingo (5.2.0).
Benchmarks were performed on a Linux machine, with two 12-core AMD Opteron 6176 SE CPUs and 128 GB RAM.
The timeout set to 300 seconds and memory limited to 8GB.
Every instance requested 10 answer sets from the solver.
Instances are available here.

Ground Explosion Benchmark. Time in seconds, timeout in parentheses. Instance size is number of elements in domain.
Instance size	Alpha	Omiga	ASPeRiXo	Clingo
8	1.37(0)	0.42(0)	5.54(0)	1.74(0)
10	1.48(0)	0.43(0)	0.02(0)	7.00(0)
12	1.46(0)	0.44(0)	0.02(0)	22.47(0)
14	1.64(0)	0.47(0)	0.02(0)	56.39(0)
16	1.60(0)	0.51(0)	0.03(0)	145.28(0)
18	1.64(0)	0.45(0)	0.03(0)	300.00(1)
20	1.83(0)	0.46(0)	0.05(0)	300.00(1)
22	1.69(0)	0.46(0)	0.05(0)	300.00(1)
24	1.57(0)	0.47(0)	0.06(0)	300.00(1)
26	1.24(0)	0.44(0)	0.06(0)	300.00(1)
28	1.29(0)	0.47(0)	0.06(0)	300.00(1)
30	1.52(0)	0.49(0)	0.07(0)	300.00(1)
50	1.31(0)	0.53(0)	0.11(0)	300.00(1)
100	1.60(0)	0.81(0)	0.21(0)	300.00(1)
300	1.58(0)	0.93(0)	0.63(0)	300.00(1)
500	2.19(0)	1.41(0)	1.06(0)	300.00(1)
1000	2.30(0)	1.66(0)	2.21(0)	300.00(1)

Graph 5-Colorability. Time in seconds, timeout in parentheses. Instance size is number of vertices / number of edges.
Instance size	Alpha	Omiga	ASPeRiXo	Clingo
10/40	1.41(0)	14.33(0)	31.10(1)	0.02(0)
20/80	1.53(0)	234.93(6)	128.79(4)	0.02(0)
30/120	1.59(0)	300.00(10)	230.23(7)	0.03(0)
40/160	2.54(0)	300.00(10)	217.17(7)	0.04(0)
50/200	2.31(0)	300.00(10)	300.00(10)	0.04(0)
75/300	2.74(0)	300.00(10)	300.00(10)	0.05(0)
100/400	4.24(0)	300.00(10)	300.00(10)	0.06(0)
200/800	9.52(0)	300.00(10)	300.00(10)	0.15(0)
300/1200	13.20(0)	300.00(10)	300.00(10)	0.27(0)
400/1600	22.54(0)	300.00(10)	300.00(10)	0.45(0)
500/2000	33.85(0)	300.00(10)	300.00(10)	0.68(0)
750/3000	67.22(0)	300.00(10)	300.00(10)	1.46(0)
1000/4000	119.94(0)	300.00(10)	300.00(10)	2.66(0)
Fixed vertices below.
50/50	1.88(0)	290.47(9)	0.24(0)	0.03(0)
50/100	2.05(0)	300.00(10)	0.45(0)	0.03(0)
50/200	2.31(0)	300.00(10)	300.00(10)	0.04(0)
50/300	74.39(2)	300.00(10)	300.00(10)	0.07(0)
50/400	253.80(8)	300.00(10)	300.00(10)	0.06(0)
50/500	168.76(4)	300.00(10)	300.00(10)	0.04(0)

Cutedge. Time in seconds, timeout in parentheses. Instance size is number of vertices / percentage of edge being present.
Instance size	Alpha	Omiga	ASPeRiXo	Clingo
100/30	12.59(0)	4.25(0)	0.78(0)	27.64(0)
100/50	11.87(0)	6.22(0)	1.79(0)	79.50(0)
200/30	22.90(0)	13.46(0)	13.29(0)	300.00(10)
200/50	45.95(0)	24.20(0)	35.18(0)	300.00(10)
300/10	16.92(0)	10.08(0)	8.54(0)	291.35(4)
300/30	59.58(0)	32.36(0)	72.09(0)	300.00(10)
300/50	300.00(10)	77.70(0)	239.55(1)	300.00(10)
500/10	62.46(0)	32.01(0)	70.38(0)	300.00(10)
500/30	300.00(10)	122.16(0)	300.00(10)	300.00(10)
500/50	300.00(10)	215.01(0)	300.00(10)	300.00(10)
400/10	40.97(0)	20.72(0)	27.61(0)	300.00(10)
400/30	300.00(10)	84.73(0)	284.71(4)	300.00(10)
400/50	300.00(10)	128.84(0)	300.00(10)	300.00(10)

Reachability. Time in seconds, timeout in parentheses. Instance size is number of vertices / multiplier of number of edges.
Instance size	Alpha	Omiga	ASPeRiXo	Clingo
1000/4	2.13(0)	1.21(0)	0.77(0)	0.11(0)
1000/8	3.19(0)	1.63(0)	2.57(0)	0.21(0)
10000/2	10.95(0)	7.82(0)	31.11(0)	0.52(0)
10000/4	13.06(0)	22.55(0)	130.00(0)	1.09(0)
10000/8	16.62(0)	56.93(0)	300.00(10)	2.27(0)

Alpha is open source and available at github. A paper preprint is available here.

Benchmarks results will be available soon.