Packetdrill -- a powerful tool for testing TCP stack behavior

Abstract:packetdrillIt is a very useful tool for testing network protocol stackGoogleDevelopment, it is often used to regression test the network protocol stack to ensure that the new function will not affect the original function. This paper mainly introduces its basic principle, installation, introduction and writing method of test script.

1. introduction

packetdrillIt is a very useful tool for testing network protocol stackGoogleDevelopment, it is often used to regression test the network protocol stack to ensure that the new function will not affect the original function. It supportsLinux, FreeBSD, OpenBSDAndNetBSDKernel. It uses scripted language to write test statements and predict protocol stack output. The official also provides many test script examples.

2. principle

packetdrillThe overall framework of is shown in the figure below

Packetdrill -- a powerful tool for testing TCP stack behavior

packetdrillThe application internally simulates a connectedRemote terminalandLocal terminal。 amongRemote terminalAs a remote channel to send messages to the local machine, we canpacketdrillApplied introversiontunDevice writeIPThe message comes from the kernel protocol stack, which is equivalent to receiving this message from the remote endIPThe message, after routing, will be sent to the protocol stack. Conversely, the direction of the kernel protocol stackRemote terminalThe message will be sent through thistunDevice backpacketdrillAt this time, we can verify the correctness of the function of the protocol stack by comparing its input.

The script file is.pktFile with suffix,packetdrillRead the file after startup,Script parserResolve each line of script statement to runtimeeventScript runnerExecute each in turnevent

3. installation

packetdrillDependentpackage: gccpythonflexbison

Download the source code from the official GitHub and compile it

> ./configure
> make

4. entry

Execute a test script

> ./packetdrill  tests/linux/fast_retransmit/fr-4pkt-sack-linux.pkt

If there is no output, it means that the script test has passed:), otherwise, it will prompt which line of script does not meet the expectation and the reason for the error respectively

For example, an error occurred when executing the following script on my machine (kernel version 4.4.0):

> ./packetdrill  tests/linux/listen/listen-incoming-ack.pkt 
tests/linux/listen/listen-incoming-ack.pkt:17: error handling packet: bad value outbound TCP option 3
script packet:  0.200000 S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
actual packet:  0.201014 S. 0:0(0) ack 1 win 29200 <mss 1460,nop,nop,sackOK,nop,wscale 7>

It indicates that the17There was an error on line, in the scriptRemote terminalExpected receiptSYNACKIn the messagewscale=6, but in the message actually receivedwscale=7

The reason for this error is that the protocol stack implementation of the kernel version suitable for the script is inconsistent with that of my native version! The kernel version is inconsistent, and some implementations of the protocol stack are inconsistent! In this case, we can simply modify the script to adapt to the kernel version we use.

5. Script language

packetdrillInstead of using a ready-made scripting language, its scripts have sometcpdumpShadow, some moresocketTrace of programming

// Test behavior when a listener gets an incoming packet that has
// the ACK bit set but not the SYN bit set.

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0

0.100 < . 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.100 > R 0:0(0) win 0

// Now make sure that when a valid SYN arrives shortly thereafter
// (with the same address 4-tuple) we can still successfully establish
// a connection.

0.200 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
0.200 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>

0.300 < . 1:1(0) ack 1 win 320
0.300 accept(3, ..., ...) = 4

I think the best way to learn this kind of script is to learn the official example. In the example, you can build the script you need by following the rules! Really have doubt still can turn over code slightly!

time stamp

Every line of a script isTimestamp + statementThe form. The time stamp indicates the execution time of the statement,packetdrillSupportAbsolute timeandRelative timeTwo formats

//Relative time, 0.1 seconds after the last script
+.1  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

//Absolute time, 0.2 seconds after the script starts running
0.200 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

Inject message into protocol stack

Time stamp followed<The statement representation of the symbol fromRemote terminalInject message into protocol stack, followed by defaultTCPThe content of the message (of course, it can also be connected to other protocols, but the complexity of the protocol stack mostly lies inTCP)

//Inject a syn message (s for SYN), the start and end serial numbers are 0, the data length is 0, the notification window size is 32792, and the options of MSS, sack and wscale are carried.
0.200 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>

Receive message from protocol stack

Time stamp followed>Statement representation of symbolsRemote terminalThe message is expected to be received from the protocol stack. The expected receiving time here is a range[ts-tolerance,ts+tolerance], tolerance timetoleranceThe default is4MS (can be changed by running parameters)

//It is expected to receive a sync message (. Indicates ACK). The ACK sequence number is 1, which carries the options of MSS, sack and wscale
0.200 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>

system call

The message statement above is standing atRemote terminalFrom the point of view, system call is standingLocal terminalYes,packetdrillThe following system calls are supported

struct system_call_entry system_call_table[] = {
    {"socket",     syscall_socket},
    {"bind",       syscall_bind},
    {"listen",     syscall_listen},
    {"accept",     syscall_accept},
    {"connect",    syscall_connect},
    {"read",       syscall_read},
    {"readv",      syscall_readv},
    {"recv",       syscall_recv},
    {"recvfrom",   syscall_recvfrom},
    {"recvmsg",    syscall_recvmsg},
    {"write",      syscall_write},
    {"writev",     syscall_writev},
    {"send",       syscall_send},
    {"sendto",     syscall_sendto},
    {"sendmsg",    syscall_sendmsg},
    {"fcntl",      syscall_fcntl},
    {"ioctl",      syscall_ioctl},
    {"close",      syscall_close},
    {"shutdown",   syscall_shutdown},
    {"getsockopt", syscall_getsockopt},
    {"setsockopt", syscall_setsockopt},
    {"poll",       syscall_poll},
    {"cap_set",    syscall_cap_set},
    {"open",       syscall_open},
    {"sendfile",   syscall_sendfile},
    {"epoll_create", syscall_epoll_create},
    {"epoll_ctl",    syscall_epoll_ctl},
    {"epoll_wait",   syscall_epoll_wait},
    {"pipe",         syscall_pipe},
    {"splice",       syscall_splice},

What’s different from the system call we are used to is that,packetdrillSome parameters in the system call in cannot be changed. We need to fill in...(script running opportunity will help us to fill in), in addition, we need to set its return value.

//The socket system call returns FD = 3, where the three are only valid in the script scope. The descriptor value returned at runtime is maintained by the framework, and the framework will maintain their corresponding relationship
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3

//Setsockopt system call, the third parameter [1] represents a pointer to the value 1
0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

//Listen system call, the last two parameters are determined by the framework
0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0


Sometimes we need to peepTCPMore information about the runtime, such asMSSWhat is the current window sizecwndWhat is the slow start thresholdssthreshHow much is it? Now we can useassertStatement to expect its state

//Expected status information at this time
0.300 %{
assert tcpi_reordering == 3
assert tcpi_unacked == 10
assert tcpi_sacked ==  1

packetdrillSupport expectationsTCPThe information is as follows:

/* packetdrill/gtests/net/packetdrill/tcp.h */
struct _tcp_info {
    __u8    tcpi_state;
    __u8    tcpi_ca_state;
    __u8    tcpi_retransmits;
    __u8    tcpi_probes;
    __u8    tcpi_backoff;
    __u8    tcpi_options;
    __u8    tcpi_snd_wscale:4, tcpi_rcv_wscale:4;
    __u8    tcpi_delivery_rate_app_limited:1;

    __u32    tcpi_rto;
    __u32    tcpi_ato;
    __u32    tcpi_snd_mss;
    __u32    tcpi_rcv_mss;

    __u32    tcpi_unacked;
    __u32    tcpi_sacked;
    __u32    tcpi_lost;
    __u32    tcpi_retrans;
    __u32    tcpi_fackets;

    /* Times. */
    __u32    tcpi_last_data_sent;
    __u32    tcpi_last_ack_sent;     /* Not remembered, sorry. */
    __u32    tcpi_last_data_recv;
    __u32    tcpi_last_ack_recv;

    /* Metrics. */
    __u32    tcpi_pmtu;
    __u32    tcpi_rcv_ssthresh;
    __u32    tcpi_rtt;
    __u32    tcpi_rttvar;
    __u32    tcpi_snd_ssthresh;
    __u32    tcpi_snd_cwnd;
    __u32    tcpi_advmss;
    __u32    tcpi_reordering;

    __u32    tcpi_rcv_rtt;
    __u32    tcpi_rcv_space;

    __u32    tcpi_total_retrans;

    __u64    tcpi_pacing_rate;
    __u64    tcpi_max_pacing_rate;
    __u64    tcpi_bytes_acked;    /* RFC4898 tcpEStatsAppHCThruOctetsAcked */
    __u64    tcpi_bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived */
    __u32    tcpi_segs_out;         /* RFC4898 tcpEStatsPerfSegsOut */
    __u32    tcpi_segs_in;         /* RFC4898 tcpEStatsPerfSegsIn */

    __u32    tcpi_notsent_bytes;
    __u32    tcpi_min_rtt;
    __u32    tcpi_data_segs_in;    /* RFC4898 tcpEStatsDataSegsIn */
    __u32    tcpi_data_segs_out;    /* RFC4898 tcpEStatsDataSegsOut */
    __u64   tcpi_delivery_rate;

    __u64    tcpi_busy_time;      /* Time (usec) busy sending data */
    __u64    tcpi_rwnd_limited;   /* Time (usec) limited by receive window */
    __u64    tcpi_sndbuf_limited; /* Time (usec) limited by send buffer */

Special attention is needed when usingassertWe need to make surestruct _tcp_infoStructure inpacketdrillIt is consistent with the definition in the current kernel. Otherwise, an error will be reported!