The v2.5.0 comparator campaign: plan¶

Status: complete (2026-10-28). The results are at the end of this page. Above them, unchanged, is the pre-registration. It fixed the cells, the rules and the reading of the v2.5.0 adaptive-routing comparators (#563) before the first run exists, as the v2.4.0 control did. The plan file is tools/bench/campaigns/v2.5.0-comparators.yaml; its jobs are dispatched and collected with tools/bench/campaign.py and read with tools/bench/comparator_view.py.

← Benchmark index · Methodology · Roadmap

What this campaign decides¶

No default. AntHocNet does not run in it. It measures three protocols that AntHocNet's own lineage is usually compared with. They live in the adaptive module (ns3/baselines/adaptive/):

arm	issue	source	kind
`ara`	#586	Günes, Sorges & Bouazizi, ICDCS-W 2002	ant-colony, on demand
`termite`	#587	Roth, PhD dissertation, Cornell 2005, ch. 4	stigmergic, pheromone on every packet
`qrouting`	#588	Boyan & Littman, NIPS 1994	learned (Q-learning on delivery time)

A comparator's numbers count only after it clears three gates (AGENTS.md, Conventions):

a fidelity sheet from its paper (the module README);
the per-PR delivery smoke (#439);
an anchor that reproduces its own paper's trend.

This campaign runs gate 3, then a family comparison for the arms that pass. An arm whose anchor fails ships with a written infeasibility verdict on its issue instead of family numbers.

Part 1: anchors¶

Ten seeds per cell, on anthocnet-compare at ns-3.42. Each rule is fixed here. A result that misses its rule is reported as a miss, with no re-run under changed settings.

ARA: pause sweep against AODV¶

The paper's setup (§4): 50 nodes, 1500 × 300 m, random waypoint at up to 10 m/s, 900 s, 10 CBR flows. Pauses are 0, 30, 120 and 600 s, out of the paper's seven. Its findings: under high mobility, ARA delivers more than 95 % while AODV does not. Below roughly 100 s of pause, ARA has the least overhead. From 300 to 600 s, AODV and ARA are close.

Rule. Paired by seed, ARA minus AODV:

PDR: positive with the 95 % CI excluding zero at pause 0 and at pause 30;
NRL: negative with the CI excluding zero at pause 0.

The 120 and 600 s cells are reported, not gated.

Termite: goodput and path length against speed¶

The trend comes from §4.4 of Roth's dissertation, which evaluates the chapter-4 Termite this arm implements. §7.3 evaluates ReTermite, a different protocol, so it is not this arm's anchor.

§4.4's setup is 100 nodes and 600 s, with zero pause after a 60 s start. Every node sends a 64-byte packet every 0.5 s and receives an acknowledgement for each. The field is 50 × 50 m at 10 m range, with a perfect MAC. ns-3 has no perfect MAC. The anchor therefore keeps the field in ranges: 1250 × 1250 m at 250 m, the same 5 × 5 ranges.

Traffic is ten pairs of 64-byte flows at 1024 bit/s, each with a reply flow standing in for the acknowledgement (fixture). Speeds are 1, 5, 10 and 20 m/s.

§4.4.2 finds that data goodput declines as mobility rises, and that delivered data packets take sharply longer paths while route request and reply paths stay short. §4.4 has no reference protocol; AODV runs alongside for context only.

Rule. Comparing 20 m/s against 1 m/s, as a difference of means with a Welch 95 % CI:

Termite's PDR falls, with the CI excluding zero;
its mean hop count over delivered data (##ROUTE##, last field) rises, with the CI excluding zero.

The 5 and 10 m/s cells are reported, not gated.

Q-routing: low against high load, against the oracle¶

Boyan & Littman's finding (§3) is that at low load Q-routing settles on the shortest paths. At high load it routes around congestion and delivers faster than shortest-path routing.

The anchor is a static field: 36 nodes in 600 × 600 m, 175 m range, pause > run time, 300 s, 10 flows. It runs at two loads, 2048 and 65536 bit/s per flow, against the shortest-hop oracle.

Rule. Paired by seed, Q-routing minus oracle:

at low load, the PDR CI lies wholly above −1 pp;
at high load, mean delay is negative with the CI excluding zero.

Part 2: family comparison¶

The family comparison follows ADR-0026: each arm runs where it is native.

ara and termite are MANET protocols. They run the six MANET grid cells.
qrouting is not tied to a family. It also runs the static mesh, FANET (main and sparse), VANET (main and sparse) and disaster cells.

The arms run alone, on the v2.4.0 control's command lines and seeds (20 per cell: 10 for each grid block, 5 × 4 for the static mesh). Their rows are then paired by seed with the committed v2.4.0 rows, results/cells/<cell>-v240.txt. That works because the v2.5.0 scale screen's recheck reproduced those rows byte for byte (see sweeps/scale.md). So a v2.5.0 arm row and a v2.4.0 row with the same seed saw the same mobility and traffic.

Each arm cell is read three ways, as arm minus reference, with bench_parse.paired_stats:

against AntHocNet;
against the cell's oracle;
against the cell's best native baseline: the one native_view.py picks (highest mean PDR, ties broken on delay99).

These readings are descriptive, with no pass/fail rule. They answer one question: where does each lineage stand against AntHocNet and the natives, and in which families?

Out of scope. The satellite (ISL grid, LEO) and SAGIN cells run on the isl-grid and leo-walker harnesses. Those harnesses do not carry the adaptive module, so the arms have no satellite numbers in v2.5.0.

Reading the results¶

python3 tools/bench/campaign.py collect tools/bench/campaigns/v2.5.0-comparators.yaml \
    --log RUNS.tsv --out results/cells
python3 tools/bench/comparator_view.py --gen v250c --control v240

comparator_view.py prints three tables:

the anchor table;
the Termite trend;
the family table.

The results section of this page, and the verdicts on #586, #587 and #588, are written from that output.

Results¶

Campaign v250c, dispatched and collected on 2026-10-28 at main, ns-3.42 -opt. The cells are committed as results/cells/<cell>-v250c.txt. Every number below is from tools/bench/comparator_view.py; differences are arm minus reference, with 95 % CIs.

Verdicts¶

arm	fidelity sheet	#439 smoke	anchor	numbers count
`ara`	✔	✔ 100.0 %	passes	yes
`termite`	✔	✔ 19.6 %	misses (the trend points the right way; its CIs span zero)	no: infeasibility verdict on #587
`qrouting`	✔	✔ 100.0 %	misses (high load)	no: infeasibility verdict on #588

Anchors¶

ARA vs AODV (10 seeds, paired):

pause	ARA PDR	AODV PDR	dPDR	d NRL	d delay99	rule
0	98.42 %	89.27 %	+9.15 [+8.44, +9.86]	−49.5 [−50.8, −48.3]	+611 ms	PDR ✔ NRL ✔
30	98.54 %	89.55 %	+8.99 [+7.06, +10.92]	−47.2 [−48.5, −45.9]	+652 ms	PDR ✔
120	98.50 %	90.24 %	+8.26 [+7.07, +9.45]	−40.5 [−42.5, −38.4]	+709 ms	reported
600	99.54 %	94.52 %	+5.02 [+3.91, +6.13]	−31.1 [−34.4, −27.8]	−196 ms	reported

The paper's trend reproduces:

ARA stays above 95 % delivery at every pause, and AODV does not reach it.
ARA has the least overhead under high mobility.

What the paper does not show is ARA's tail delay: 0.6 to 0.7 s higher at short pauses. Packets wait while the BANT retraces the FANT's path, and again when they are re-routed after a MAC or ARP drop.

Q-routing vs the shortest-hop oracle (10 seeds, paired):

load	Q-routing PDR	oracle PDR	dPDR	d mean delay	rule
2048 bit/s/flow	99.99 %	99.65 %	+0.34 [−0.45, +1.13]	—	PDR CI above −1 pp ✔
65536 bit/s/flow	27.20 %	49.74 %	−22.54 [−25.93, −19.15]	+158 ms [+65, +251]	mean delay below 0 ✘

The low-load half of Boyan & Littman's trend holds: Q-routing settles on shortest paths. The high-load half does not. Their gain comes from routing around queues in a wired network. Here the shared channel is past saturation: the oracle itself loses 49.7 % of packets on the channel and Q-routing 73.4 %, so a detour adds transmissions to the contended medium instead of relieving a queue.

Per the rule, the arm ships with this verdict and its family numbers do not count. The six Q-routing-only family cells (static mesh, FANET ×2, VANET ×2, disaster) were withdrawn before dispatch. The learned comparator that is designed for a wireless channel is #756.

Termite: speed trend (Roth §4.4; 10 seeds per speed, two-way flows):

speed	Termite PDR	Termite data hops	AODV PDR (context)	AODV hops
1 m/s	47.45 %	6.65	78.25 %	3.11
5 m/s	45.43 %	6.96	72.79 %	3.17
10 m/s	43.75 %	7.16	70.91 %	3.11
20 m/s	42.54 %	7.94	67.05 %	3.13

Rule: 20 m/s minus 1 m/s, Welch 95 % CI.

metric	difference	rule	verdict
PDR	−4.91 pp [−10.36, +0.54]	must fall, CI excluding 0	✘
data hops	+1.29 [−0.38, +2.96]	must rise, CI excluding 0	✘

Both quantities move the way §4.4 reports, monotonically at every speed step. Neither difference separates from zero at ten seeds, so the anchor is a miss. As pre-registered, there is no re-run with more seeds.

The arm ships with this verdict, and its family numbers below do not count. Two things limit the reproduction:

ns-3 has no perfect MAC, which §4.4's goodput-against-speed curve relies on.
The per-packet acknowledgement is modelled as a reply flow.

Seed-to-seed spread is large: Termite's PDR SD is 7.2 pp at 1 m/s and 3.6 pp at 20 m/s.

Family comparison: the MANET grid¶

Each arm runs alone on the v2.4.0 control's command lines and seeds (20 per cell). Its rows are paired by seed with the committed <cell>-v240.txt rows.

PDR, % (20-seed mean):

cell	ara	termite¹	qrouting¹	AntHocNet	best native	oracle
grid-rwp-tworay	99.37	66.28	94.38	98.47	88.30 (OLSR)	97.94
grid-ssrwp-tworay	99.48	65.42	94.86	98.47	89.16 (OLSR)	98.02
grid-gaussmarkov-tworay	98.55	64.20	88.45	97.11	85.84 (AODV)	96.55
grid-rwp-nakagami²	88.17	37.19	94.40	91.90	85.72 (OLSR)	96.09
grid-ssrwp-nakagami²	88.44	37.19	94.67	90.66	84.91 (OLSR)	95.55
grid-gaussmarkov-nakagami²	80.05	36.04	91.53	86.86	78.28 (OLSR)	94.82

¹ Reported for completeness. The arm's anchor missed, so these numbers do not count. ² Fading channel. Until v2.6.0 the drop-cause breakdown of ara and termite did not close here: packets re-routed after a loss were counted once per lost hop (#761). The committed files were re-measured with the per-packet book on 2026-10-29. Every ##RUN##, ##ROUTE## and ##COMMON## row reproduced byte for byte, and only the drop columns changed. The books now close at 99.1 to 99.6 %: - ara loses mostly to route failures: 8.5 to 15.0 % of offered packets. - termite loses about 30 % at the MAC and about 30 % on the channel. - qrouting loses about 5 % to TTL.

ARA minus AntHocNet, paired:

cell	dPDR	d delay99	d NRL
grid-rwp-tworay	+0.90 [+0.74, +1.06]	+887 ms	−5.39 [−5.49, −5.28]
grid-ssrwp-tworay	+1.00 [+0.81, +1.20]	+889 ms	−5.24 [−5.34, −5.14]
grid-gaussmarkov-tworay	+1.44 [+1.19, +1.69]	+865 ms	−6.20 [−6.38, −6.01]
grid-rwp-nakagami	−3.72 [−4.79, −2.66]	+9.0 s	−6.64 [−7.08, −6.20]
grid-ssrwp-nakagami	−2.23 [−3.44, −1.01]	+8.9 s	−6.94 [−7.42, −6.46]
grid-gaussmarkov-nakagami	−6.81 [−7.75, −5.88]	+9.4 s	−7.56 [−8.52, −6.60]

Termite minus AntHocNet, paired:

cell	dPDR
tworay cells	−32.2 to −33.1 pp
nakagami cells	−50.8 to −54.7 pp

What it says:

ARA is the strongest comparator AntHocNet has met on the grid.
On the two-ray cells it delivers 0.9 to 1.4 pp more, at a fifth of AntHocNet's routing overhead (NRL about 1.6 against about 7).
It pays for that with a 99th-percentile delay near 0.9 s higher, the cost of holding packets for its route discovery.
On a fading (Nakagami) channel ARA loses: 2.2 to 6.8 pp less delivery and tail delays of 9 s. AntHocNet's proactive maintenance and multipath keep working when links flicker, where ARA's purely reactive repair does not.
ARA beats the best native protocol in every cell (+1.8 to +12.7 pp).
Termite (not counted: its anchor missed) trails every protocol on the grid under one-way CBR. Its data walks until a seed or a reply lays the destination's trail. On the two-ray cells 29 to 31 % of data packets die on TTL; on the Nakagami cells about 30 % die at the MAC and 30 % on the channel (the per-packet book, #761). This is what chapter 4 of Roth's dissertation implies for silent sinks; it is not an implementation defect (see the anchor above for the two-way regime).