How to run 128 (testnet) I2P routers in multiple subnets on a single Linux system.

2018-10-16 13:36:20 +0200 - Written by Mikal Villa

For a long time, at least internally it’s been talking about the need of a testnet for I2P. Testing in production isn’t trivial :)

I’ve been on and off the mission in quiet for myself for a while now, but finally completed something worty publishing, in other words a working testnet setup/teardown script. When I was thinking about the case two technologies comes to mind which might help getting the testnet idea an reality is cgroups(namespaces) in Linux and rdomains in OpenBSD - both of them for network isolation and virtualization. It’s probably more as well, but it’s the ones I know of by the time of writing.

Currently I’m running a 128x node testnet divided on ~7x subnets and 8x virtual switches all on the same Linux kernel/machine.

Github link to my script files is here.

Most likely, this is the first post in a series of the topic I2P testnet.

Some quick FAQs/General information:

  • Why didn’t I use docker?
    • Bloated in this user-case.
    • Faster to update one binary, then building an container.
    • Only the network part needs to be isolated as long as routers has custom data/config directory.
  • Kubernetes support?
    • Chill, it’s 0.0.1 alpha.
    • It’s coming :)
    • Yes, it means it has to support docker/containers as well in the end.
  • Information collection?
    • TBA.
    • For now, manual curl/browser to namespaces.
  • Why i2pd and not java i2p?
    • Bigger network for less resources.
    • Will of course support java quite soon.
  • I’m getting errors, how can I debug this shit?
    • Change the shebang of the script from #!/usr/bin/env bash to #!/usr/bin/env bash -x
  • Script hacking notes at the bottom of the post.

What are the dependencies for now?

  • i2p-tools (reseed server) needs to be resolvable in $PATH
    • Built in golang, you’ll need it to compile.
  • i2pd needs to be resolvable in $PATH
    • libboost.
    • openssl.
    • (optional) libminiupnpc.
  • Linux kernel with cgroups enabled.
    • I doubt you got a linux system not supporting it, don’t worry.

What are the steps to create a testnet?

  • We’ll need some initial nodes.
  • And a reseed webserver to host the routerInfo’s for them.
  • I’ve added mine routerinfo’s in a tar.xz in the github repo if you want to skip this part.
  1. Boot up the network.
  2. From the git repo, use collect-routerinfos.sh to move the created router.info files into the reseed/netDb directory with a routerInfo-(whatever).dat format.
  3. Restart the network so it can bootstrap.
  4. Wait for the routers to connect and create tunnels.
  5. Enjoy a coffee. :)

Usage?

  • Use ./i2ptestnet.sh -c testnet.conf to only clean (teardown the testnet and clean the host)
  • Use ./i2ptestnet.sh testnet.conf to initialize a testnet (no need to run the clean command up front since it will check & remove any existing related setup before it starts)

So, how does the script work?

First of, it’s a clean script, meaning it can clean up after itself, and it also ensures to bring all down before up, to avoid any hanging-leftovers from earlier runs to ruin the current.

It parses a config file in a format of an simple DSL. It gives you the posibility to split your testnet into several subnets instead of running all of your testnodes in the same which is lame - and it makes it less hassle to connect with a kubernetes cluster in big scale. :)

I cut some of the DSL out bellow, basically I’ve defined several networks with a “switch” type to link them, and “host” types for defining “fake i2p routers” - both types represent a network namespace. It also support running commands in a network namespace via the “exec” subcommand of either “switch” or “host”.

#### Special cases ####

host fw
  dev veth0 10.0.0.254/24
  dev veth1 10.1.1.254/24
  dev vbr0eth2
  dev vbr0eth3
  dev vbr1eth4
  dev vbr1eth5
  dev vbr2eth6
  dev vbr2eth7
  dev vbr3eth8
  dev vbr3eth9
  dev vbr4eth10
  dev vbr4eth11
  bridgedev vbr0 vbr0eth2 vbr0eth3 10.23.23.254/24
  bridgedev vbr1 vbr1eth4 vbr1eth5 10.45.45.254/24
  bridgedev vbr2 vbr2eth6 vbr2eth7 192.168.24.1/24
  bridgedev vbr3 vbr3eth8 vbr3eth9 10.78.17.254/24
  bridgedev vbr4 vbr4eth10 vbr4eth11 10.78.100.254/24
  route default via 10.1.1.253
  exec echo 1 > /proc/sys/net/ipv4/ip_forward
  i2preseed

host gw
  dev veth0 fw/veth1 10.1.1.253/24
  dev veth1 192.168.1.254/24
  dev veth2 192.168.2.254/24
  route default via 10.1.1.254
  exec echo 1 > /proc/sys/net/ipv4/ip_forward

#### Normal Hosts ####

host host01
  dev veth0 10.0.0.1/24
  route default via 10.0.0.254
  i2pdnode 15

host host02
  dev veth0 10.0.0.2/24
  route default via 10.0.0.254
  i2pdnode 16

.....


host host126test
  dev veth0 10.78.100.26/24
  route default via 10.78.100.254
  i2pdnode 126

host host127test
  dev veth0 10.78.100.27/24
  route default via 10.78.100.254
  i2pdnode 127

host host128test
  dev veth0 10.78.100.28/24
  route default via 10.78.100.254
  i2pdnode 128

###### Switches #########

switch sw0
  dev d01 fw/veth0
  dev d02 host01/veth0
  dev d03 host02/veth0
  dev d04 fw/vbr2eth6
  dev d05 fw/vbr3eth8
  dev d06 fw/vbr4eth10

switch sw2
  dev d01 fw/vbr0eth2
  dev d02 host21/veth0
  dev d03 host22/veth0
  dev d04 host23/veth0
  dev d05 host24/veth0

switch sw3
  dev d01 fw/vbr0eth3
  dev d02 host31/veth0
  dev d03 host32/veth0
  dev d04 host33/veth0
  dev d05 host34/veth0
  
.....

The script itself is “just” about 500 lines of code, which also is pasted bellow.

#!/usr/bin/env bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"

DATADIR=${DATADIR:-"$DIR/data"}
PIDDIR=${PIDDIR:-"$DATADIR/pids/"}
LOGDIR=${LOGDIR:-"$DATADIR/logs/"}

RESEEDSERVER=${RESEEDSERVER:-"https://192.168.24.1:8443/"}
RESEEDSERVER_IP=${RESEEDSERVER_IP:-"192.168.24.1"}
RESEED_DIR="$DATADIR/reseed"
NETDB_DIR="$DATADIR/reseed/netDb"
SIGNER=${SIGNER:-"meeh@mail.i2p"}
SIGNER_FNAME=${SIGNER_FNAME:-"meeh_at_mail.i2p"}

mkdir -p $DATADIR $PIDDIR $LOGDIR

# Tmux session for all routers
tmux new-session -d -s routers top

I2PDWRAPPER=$(cat <<EOF
#!/usr/bin/env bash\n
ip netns exec \$2 i2pd \
  --datadir=$DATADIR/testnode\$1 \
  --log stdout \
  --floodfill \
  --ipv4=1 \
  --host=\$3 \
  --ifname4=veth0 \
  --pidfile=$PIDDIR/router\$1.pid \
  --port=4764 \
  --reseed.urls=$RESEEDSERVER | tee -a $LOGDIR/router\$1.log >> $LOGDIR/all-routers.log &
EOF
)


for i in `seq 1 128`; do
  mkdir -p $DATADIR/testnode$i/certificates/reseed
  cp $DATADIR/reseed/${SIGNER_FNAME}.crt $DATADIR/testnode$i/certificates/reseed/
  echo -e $I2PDWRAPPER > $DATADIR/testnode$i/i2pdwrapper.sh
done

#echo $I2PDWRAPPER
#exit 0


haderror=""

if [ "`id -r -u`" != "0" ]
then
  echo "Must be root"
  echo ""
  haderror="Y"
fi

if [ "$haderror" -o $# -lt 1 ]
then
  echo "Usage: sudo ./i2ptestnet.sh [-c] setup-file"
  exit 1
fi

if [ "$1" = "-c" ]
then
  cleanonly=Y
  shift
else
  cleanonly=
fi

setup="$1"
shift
setupbase="$(basename $setup)"
errline=""
error=""

if ! MYTMP="`mktemp -d -t i2ptestnet-XXXXXX`"
then
            echo >&2
            echo >&2
            echo >&2 "Cannot create temporary directory."
            echo >&2
            exit 1
fi

myexit() {
  status=$?
  if [ "$error" != "" ]
  then
    echo "$setupbase: line $errline: $error"
  fi
  rm -rf $MYTMP
  exit $status
}

trap myexit INT
trap myexit HUP
trap myexit 0

CURDIR=`pwd`/
export CURDIR

set -e

mkdir $MYTMP/setup
(echo "cd $CURDIR"; sed = "$setup" | sed -e 'N;s/\n/\t/' -e 's/^/lineno=/' -e '/exec/s/[<>|&]/\\&/g' -e '/i2preseed/s/[<>|&]/\\&/g' -e '/i2pdnode/s/[<>|&]/\\&/g') > $MYTMP/setup/$setupbase

mkdir $MYTMP/ns
mkdir $MYTMP/runtime-lines

current_name=

create_namespace() {
  errline=$lineno
  local type="$1"
  current_name="$2"
  NSTMP=$MYTMP/ns/$current_name
  if [ -d $NSTMP ]
  then
    error="$current_name: $(cat $NSTMP/type) already defined"
    return 1
  fi
  mkdir $NSTMP
  mkdir $NSTMP/devices
  mkdir $NSTMP/devicepairs
  echo $type > $NSTMP/type
  echo 0 > $NSTMP/forward
  > $NSTMP/routes
  > $NSTMP/devlist
  > $NSTMP/pairlist
  > $NSTMP/bridgelist
  echo $current_name >> $MYTMP/nslist
  echo $errline > $MYTMP/runtime-lines/$current_name
}

host() {
  errline=$lineno
  create_namespace host "$1"
}

switch() {
  errline=$lineno
  create_namespace switch "$1"
}

dev() {
  errline=$lineno
  device="$1"
  shift

  if [ ! "$current_name" ]
  then
    error="cannot define dev outside of a host or switch"
    return 1
  fi

  if [ -f $NSTMP/devices/$device ]
  then
    error="$current_name/$device: already defined"
    return 1
  fi

  local otherns=
  local otherdev=
  case $1 in
    */[a-zA-Z@]*)
      otherns=$(echo $1 | cut -f1 -d/)
      otherdev=$(echo $1 | cut -f2 -d/)
      shift
      if [ -f $MYTMP/ns/$otherns/devicepairs/$otherdev ]
      then
        error="$otherns/$otherdev: already has paired device"
        return 1
      fi
    ;;
  esac

  local type="$(cat $NSTMP/type)"
  if [ "$*" != "" -a "$type" = "switch" ]
  then
    error="device in switch may not specify an IP address"
    return 1
  fi


  f=$NSTMP/devices/${device%@*}
  > $f
  for ip in "$@"
  do
    case $ip in
      */*)
       echo "$ip" >> $f
      ;;
      *)
       error="IP address should be expressed as ip/mask"
       return 1
      ;;
    esac
  done

  if [ "$otherdev" ]
  then
    echo "$current_name ${device%@*}" > $MYTMP/ns/$otherns/devicepairs/${otherdev%@*}
    echo "n/a n/a" > $NSTMP/devicepairs/${device%@*}
    echo "$otherns ${otherdev%@*}" >> $NSTMP/pairlist
    echo $errline > $MYTMP/runtime-lines/$otherns-pair-${otherdev%@*}
  fi

  echo ${device%@*} >> $NSTMP/devlist
  echo $errline > $MYTMP/runtime-lines/$current_name-dev-${device%@*}
  return 0
}

route() {
  errline=$lineno
  if [ ! "$current_name" ]
  then
    error="can only specify route in a host"
    return 1
  fi

  local type="$(cat $NSTMP/type)"
  if [ "$type" = "switch" ]
  then
    error="can only specify route in a host"
    return 1
  fi

  echo "$*" >> $NSTMP/routes
  echo $errline >> $MYTMP/runtime-lines/$current_name-routes
  return 0
}

bridgedev() {
  errline=$lineno
  device="$1"
  shift

  if [ ! "$current_name" ]
  then
    error="can only specify bridgedev in a host"
    return 1
  fi

  local type="$(cat $NSTMP/type)"
  if [ "$type" = "switch" ]
  then
    error="can only specify bridgedev in a host"
    return 1
  fi

  if [ -f $NSTMP/devices/${device%@*} ]
  then
    error="$current_name/${device%@*}: already defined"
    return 1
  fi

  ipf=$NSTMP/devices/${device%@*}
  devf=$ipf-bridged
  > $ipf
  > $devf
  for ipordev in "$@"
  do
    case $ipordev in
      */*)
       echo "$ipordev" >> $ipf
      ;;
      *)
       echo "$ipordev" >> $devf
      ;;
    esac
  done

  echo ${device%@*} >> $NSTMP/bridgelist
  echo $errline > $MYTMP/runtime-lines/$current_name-dev-${device%@*}
  return 0
}

exec() {
  errline=$lineno
  if [ ! "$current_name" ]
  then
    error="can only specify exec in a host or switch"
    return 1
  fi

  echo "$*" >> $NSTMP/exec
  echo $errline >> $MYTMP/runtime-lines/$current_name-exec
  return 0
}

i2pdnode() {
  errline=$lineno
  if [ ! "$current_name" ]
  then
    error="can only specify i2pdnode in a host or switch"
    return 1
  fi

  echo -e "$*" >> $NSTMP/i2pdnodeid
  echo $errline >> $MYTMP/runtime-lines/$current_name-i2pdnodeid
  return 0
}


i2preseed() {
  errline=$lineno
  if [ ! "$current_name" ]
  then
    error="can only specify i2preseed in a host or switch"
    return 1
  fi

  echo -e "$*" >> $NSTMP/i2preseed
  echo $errline >> $MYTMP/runtime-lines/$current_name-i2preseed
  return 0
}

cd $MYTMP/setup
. $setupbase
errline=""
cd $CURDIR

exists_ns() {
  if [ "$(ip netns list | grep "^$1\$")" ]
  then
    return 0
  else
    return 1
  fi
}

dev_in_ns() {
  ip netns exec $1 ip link list | grep "^[0-9]" | cut -d: -f2 | tr -d ' '
}

get_pids() {
  # Not in all versions:
  #   ip netns pids $1
  find -L /proc/[0-9]*/ns -maxdepth 1 -samefile /var/run/netns/$1 2>/dev/null | cut -f3 -d/
}

shutdown_ns() {
  for i in $(dev_in_ns $1)
  do
    ip netns exec $1 ip link set ${i%@*} down
  done
  pids=$(get_pids $1)
  if [ "$pids" ]; then kill $pids; sleep 1; fi
  pids=$(get_pids $1)
  if [ "$pids" ]; then kill -9 $pids; fi
}

startup_ns() {
  for i in $(dev_in_ns $1)
  do
    ip netns exec $1 ip link set ${i%@*} up
  done
}

while read ns
do
  while read dev
  do
    read errline < $MYTMP/runtime-lines/$ns-dev-$dev
    if [ ! -f $MYTMP/ns/$ns/devicepairs/$dev ]
    then
      error="$ns/$dev has no paired device"
      exit 1
    fi
  done < $MYTMP/ns/$ns/devlist

  while read otherns otherdev
  do
    read errline < $MYTMP/runtime-lines/$otherns-pair-$otherdev
    if [ ! -f $MYTMP/ns/$otherns/devices/$otherdev ]
    then
      error="$otherns/$otherdev not defined to be paired with"
      exit 1
    fi
  done < $MYTMP/ns/$ns/pairlist
done < $MYTMP/nslist

while read ns
do
  read errline < $MYTMP/runtime-lines/$ns
  error="shutting down namespace: $ns"
  exists_ns $ns && shutdown_ns $ns
done < $MYTMP/nslist

while read ns
do
  read errline < $MYTMP/runtime-lines/$ns
  error="deleting namespace"
  exists_ns $ns && ip netns del $ns
done < $MYTMP/nslist

if [ "$cleanonly" ]
then
  error=""
  exit 0
fi

while read ns
do
  read errline < $MYTMP/runtime-lines/$ns
  error="adding namespace"
  type="$(cat $MYTMP/ns/$ns/type)"
  ip netns add $ns
  if [ "$type" = "switch" ]
  then
    error="adding bridge to switch namespace"
    ip netns exec $ns brctl addbr switch
  fi
done < $MYTMP/nslist

while read ns
do
  type="$(cat $MYTMP/ns/$ns/type)"
  while read dev
  do
    read errline < $MYTMP/runtime-lines/$ns-dev-${dev%@*}
    read ons odev < $MYTMP/ns/$ns/devicepairs/${dev%@*}
    if [ "$ons" != "n/a" ]
    then
      error="adding virtual ethernet to $type namespace"
      ip link add ${dev%@*} netns $ns type veth peer netns $ons name ${odev%@*}
    else
      : # gets set up from the other end
    fi
    if [ "$type" = "switch" ]
    then
      error="adding virtual ethernet to bridge"
      ip netns exec $ns brctl addif switch ${dev%@*}
    fi
    while read ip
    do
      error="adding ip address to virtual ethernet"
      ip netns exec $ns ip addr add $ip dev ${dev%@*}
    done < $MYTMP/ns/$ns/devices/${dev%@*}
  done < $MYTMP/ns/$ns/devlist

  while read bridge
  do
    read errline < $MYTMP/runtime-lines/$ns-dev-$bridge
    error="adding bridge to host namespace"
    ip netns exec $ns brctl addbr $bridge
    while read dev
    do
      error="adding virtual interface to bridge"
      ip netns exec $ns brctl addif $bridge ${dev%@*}
    done < $MYTMP/ns/$ns/devices/$bridge-bridged
    while read ip
    do
      error="adding ip to virtual interface"
      ip netns exec $ns ip addr add $ip dev $bridge
    done < $MYTMP/ns/$ns/devices/$bridge
  done < $MYTMP/ns/$ns/bridgelist
done < $MYTMP/nslist

while read ns
do
  read errline < $MYTMP/runtime-lines/$ns
  error="starting namespace"
  startup_ns $ns

  while read route
  do
    errline=$(tr "\n" "/" < $MYTMP/runtime-lines/$ns-routes | sed -e s:/$::)
    error="adding route to $ns"
    ip netns exec $ns ip route add $route
  done < $MYTMP/ns/$ns/routes

  if [ -f $MYTMP/ns/$ns/exec ]
  then
    errline=$(tr "\n" "/" < $MYTMP/runtime-lines/$ns-exec | sed -e s:/$::)
    error="running exec for $ns"
    ip netns exec $ns sh -e $MYTMP/ns/$ns/exec
  fi

  if [ -f $MYTMP/ns/$ns/i2pdnodeid ]
  then
    errline=$(tr "\n" "/" < $MYTMP/runtime-lines/$ns-i2pdnodeid | sed -e s:/$::)
    error="running i2pdnode for $ns"
    routerId="`cat $MYTMP/ns/$ns/i2pdnodeid`"
    #echo "Router id to handle is $routerId"
    routerAddress="`ip netns exec $ns ip -o addr show veth0 | grep -v inet6 | awk '{ print $4 }' | sed 's#/24##'`"
    ip netns exec $ns bash $DATADIR/testnode$routerId/i2pdwrapper.sh $routerId $ns $routerAddress
    #i2pd \
    #  --datadir=$DATADIR/testnode$routerId \
    #  --log file \
    #  --daemon \
    #  --floodfill \
    #  --ipv4=1 \
    #  --host=$routerAddress \
    #  --ifname=veth0 \
    #  --logfile=$LOGDIR/router$routerId.log \
    #  --pidfile=$PIDDIR/router$routerId.pid \
    #  --port=4764 \
    #  --reseed.urls=$RESEEDSERVER || {
    #    echo "Can't start i2pd, total fucking error!"
    #    exit 1
    #  }
  fi

  if [ -f $MYTMP/ns/$ns/i2preseed ]
  then
    errline=$(tr "\n" "/" < $MYTMP/runtime-lines/$ns-i2preseed | sed -e s:/$::)
    error="running i2preseed for $ns"
    cd $RESEED_DIR
    #i2p-tools keygen --tlsHost $RESEEDSERVER_IP
    #i2p-tools keygen --signer $SIGNER
    echo "Setting up reseed"
    ip netns exec $ns i2p-tools reseed \
        --numRi 20 \
        --key $RESEED_DIR/${SIGNER_FNAME}.pem \
        --netdb $RESEED_DIR/netDb \
        --tlsHost $RESEED_DIR/${RESEEDSERVER_IP} \
        --tlsCert $RESEED_DIR/${RESEEDSERVER_IP}.crt \
        --tlsKey $RESEED_DIR/${RESEEDSERVER_IP}.pem \
        --signer $RESEED_DIR/${SIGNER_FNAME} &
    cd -
  fi
done < $MYTMP/nslist
error=""

while read ns
do
  echo "---------------------- $ns --------------------"
  ip netns exec $ns ip addr show
  ip netns exec $ns ip route show
  ip netns exec $ns brctl show
  echo ""
done < $MYTMP/nslist

exit 0

Script hacking/implementation notes:

  • The max 128 routers is just a hardcoded value at line 36 and can be changed. However you don’t need to adjust if you run less nodes.
  • For i2pd arugments, find the I2PDWRAPPER variable in the top of the script.
  • I learned a new bash skill, ${device%@*} will from for example with veth1@if1 remove @ and whatever that’s behind it resulting in veth1. % can also be used with . for file extension magic and so on.

Other tools I’ve found handy are OpenVSwitch and Mininet. OpenVSwitch gets really handy once we start talking about clusters and scaling horizonal when single-box setups isn’t enough anymore. It’s for example used behind the scenes in the network component of the Openstack cloud platform.