482 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 1, FEBRUARY 2017
delay of a link or a path, in a coordinated streaming environ-
ment. RLI [17] obtains per-flow delay by performing linear
interpolation based on the data collected from the reference
packets being injected to the sender. Besides the end-to-end
delay measurement approaches, there are also approaches
aiming at reconstructing per-hop delay. A recent work [9]
proposes a very effective algorithm to reconstruct additive link
metrics (e.g., delay) by path measurements. It assumes the link
metrics are constant during the measurement, which is not the
case in wireless ad-hoc networks. Another work [18] measures
and analyzes the single-hop packet delay on an IP backbone
network. It requires global clock synchronization for all routers
in the networks. These works cannot be directly adopted to
wireless ad-hoc networks due to several reasons, such as the
constrained resource and the lack of global synchronization.
Domo overcomes this difficulty by performing per-hop per-
packet delay reconstruction. At the meantime, Domo does
not depend on synchronized global clock, which is usually
not available in wireless ad-hoc networks [7] due to its high
message and energy overhead.
B. Delay Measurement in Wireless Ad-Hoc Network
End-to-End Delay: In wireless ad-hoc networks, there
are several recent works [6], [7] focusing on recording
the end-to-end delay of each collected packet at the sink.
Wang et al. [7] propose to timestamp packets at the MAC layer
to precisely record packet sending/receiving. Then the packet
sojourn time at each hop can be accumulated and recorded in
a particular field in the packet. When the packet reaches the
sink, that field will store the whole end-to-end delay. Different
from these approaches, Domo aims at reconstructing more
fine-grained delay, i,e., per-hop per-packet delay.
Per-Hop Delay: There are also several approaches [15], [16]
related to fine-grained delay reconstruction in wireless ad-hoc
networks. MNT [15] is the state-of-the-art approach to recon-
struct per-hop packet arrival order in sensor networks. For each
packet p, MNT is able to infer the two local packets right
before and after packet p at each hop. Since the generation
time of each local packet is known, the per-hop delay of
the packet p can be bounded by the two corresponding local
packets at each hop. Further, the bounds can be improved by
correlating information from packets passing through the same
forwarding nodes as packet p. The delay estimation accuracy
of MNT mainly depends on the number of packets passing
through each forwarding node per unit time, which further
mainly depends on the inter-packet interval (IPI). Therefore,
increasing the timestamping granularity of MNT cannot
improve the delay estimation accuracy. MessageTracing [16]
records every packet sent and received into the local storage
of each node. It does not reconstruct per-hop per-packet delay
directly. However, by the information it records, we can infer
the order of many packet sending/receiving events. From the
perspective of revealing the internal behaviors of the networks,
MessageTracing provides another possible method (i.e., local
logging). In the evaluation section, we will show that Domo
significantly outperforms these related approaches in terms of
the reconstruction accuracy.
III. N
ETWORK MODEL
In this section, we give the network model of this work.
We also summarize the assumptions made and notations used
in Domo. We assume that Domo works in a wireless ad-hoc
network running data collection task to a single sink node.
Each node generates and sends data packets periodically to the
sink. All nodes in the network can forward packets for other
nodes. The network includes common phenomena in wireless
ad-hoc networks such as routing dynamics, packet loss and no
global timing information.
A. Modeling
FIFO Send Queue: Each node includes an FIFO send
queue for all outgoing packets. This queue keeps the packets
generated by the node as well as packets to be forwarded. The
node will keep sending the same packet at the queue head till
a successful acknowledgement or the maximum number of
retransmission is reached. FIFO send queue is widely used
in routing protocols (e.g., CTP [19], MiniRoute) for wireless
ad-hoc networks. Note that the FIFO send queue does not
guarantee that the packet arrival order at sink is the same as the
packet arrival order at each node. In fact, existing works [15]
have shown that packets could be reordered in the network
due to the dynamic routing in sensor networks.
Routing Path Information: Since the wireless network topol-
ogy is usually dynamic. Per-hop delays are only useful when
the packet path is already known. Thanks to the recent progress
in path reconstruction [15], [20], [21], we are able to recon-
struct the packet path with small overhead, even in large scale
networks with hundreds of nodes. For example, a recent path
reconstruction approach Pathfinder [20] achieves above 96%
path reconstruction ratio in a network with 900 nodes. In case
of imperfect path information, i.e., the routing paths of some
received packets cannot be recovered, the per-hop delays
of those packets are not defined. In fact, a packet with-
out routing path information is viewed as a lost packet in
Domo. As we will show in the evaluation, Domo is able
to achieve satisfactory performance even with severe packet
losses.
Node Delay: Since wireless signal propagates very fast in
the air, a packet delay from the source to the sink actually
consists of the sojourn times the packet stays on the source
node and all forwarding nodes. The sojourn time is the time
between the node’s radio starts to receive the packet and starts
to transmit the packet. Since the time a node starts to transmit
a packet is very close to the time the next hop starts to receive
the packet. In the rest of this paper, we will use the difference
between the time a node A starts to receive a packet and the
time the next hop B starts to receive the packet to be the
node delay of this packet at node A. The sojourn time at
each node can be measured accurately on that node. In the
implementation section, we will give a detailed description of
measuring the node delay at that node.
End-to-End Delay: Recent works [6], [7] show that the
end-to-end delay can be obtained accurately after a packet
reaches the sink. MAC layer timestamping technique is used
to record the actual time a packet is transmitted by the radio